fluentd
fluentd copied to clipboard
fluent
Use zstandard for buffer compression
Is your feature request related to a problem? Please describe.
A relative newcomer to the compression scene, zstandard is at default settings superior to gzip in compression ratio, compression speed, AND decompression speed owing to it's highly optimized implementation on modern CPUs.
Adding it as an option for buffer chunk compression (with the goal of eventually making it the default) would be a pure win resulting in a reduction in buffer space usage and cpu utilization compared to using gzip. Typically applications/systems that switch to using zstd observe that reading compressed data into memory from disk is actually faster than reading uncompressed data into memory as the streaming decompression algorithm is actually faster than data can be read, even using fast SSDs. This may result in eventually it being worth it to use zstd-compression as the default even over uncompressed buffers.
Various benchmarks:
- https://community.centminmod.com/threads/round-4-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.18669/
- https://sysdfree.wordpress.com/2020/01/04/293/
- https://etbe.coker.com.au/2020/06/06/comparing-compression/
Describe the solution you'd like
This appears to be the most widely used and best-maintained Ruby library for integrating zstd compression into Ruby applications. Zstandard is also compatible with ARM and Windows platforms so it's supported everywhere Fluentd is supported today.
Describe alternatives you've considered
N/A
Additional context
No response
