Feature request: support buffer pooling

[bill-poole]

The LZ4Stream constructor accepts a "blockSize" parameter (defaulting to 1MB), which I assume results in an internal buffer of that size being created. If so, it would be great if an overload to this constructor could be provided where a buffer (byte array) could be provided to the LZ4Stream to use. This would allow us to allocate the buffer from a buffer pool (e.g. the ArrayPool<T> class in the System.Buffers library) and return the buffer to the pool once the LZ4Stream is disposed. This would greatly reduce the stress on the garbage collector.

[bill-poole]

Having now looked through the LZ4Stream.cs source code, I suspect the best way to make use of a buffer pool is to pass a buffer pool into the LZ4Stream constructor so that when it needs to allocate a new buffer, it can uses the given buffer pool.

It would be important for the LZ4Stream class to support different buffer pool implementations, which could be achieved by defining an IBufferPool interface in the LZ4 library with the following two methods:

• Allocate(int bufferSize)
• Release(byte[] buffer)

The LZ4Stream class would then accept an IBufferPool optional parameter. If non-null, the LZ4Stream class would use the buffer pool to allocate any buffers it requires, releasing the buffers back to the pool as they are no longer needed (including on LZ4Stream.Dispose).

Using a buffer pool for internal LZ4Stream buffers not only vastly reduces strain on the garbage collector, it also greatly improves memory locality (because buffers are reused), which makes much better use of the CPU caches.

The performance impact is significant when using the LZ4Stream for compressing/decompressing moderately-sized data many times (rather than one long stream of data) - such as compressing/decompressing messages.

I've tested it out and found that the aggregate throughput I observed compressing and then decompressing a 100KByte byte array 50,000 times doubled on a single core from 200MByte/s to 400MByte/sec.

[MiloszKrajewski]

Please note, I will be wokring on this in NEXT version of K4os.Compression.LZ4. Please migrate to new library.