redpanda: implement support topic compression

[dotnwat]

We accept topic compression type settings when creating topics, but we do not compress data on behalf of the producer. Therefore we implement only producer compression.type but accept any setting for compression type.

[hassila]

Would be interesting to be able to compare this at some point with producer-side compression for a use case I'm evaluating - is there any way to understand where this is on the roadmap approximately?

Basically I'm looking at a cpu-sensitive producer where we may be better of doing the I/O and let Redpanda do the compression - but only way to know for sure is to measure and contrast...

[emaxerrno]

there is a micro benchmark that tell you the time it takes to round-trip a payload in the /bench of the compression tree. you can run. it tells you even per payload size the overhead.

[emaxerrno]

@hassila https://github.com/redpanda-data/redpanda/blob/dev/src/v/compression/tests/zstd_stream_bench.cc#L51-L54

all the compressors have the same interface, so you can do snappy, zstd, etc in the redpanda code w/ the redpanda allocator there. it will print it on the terminal so you can know exactly the overhead on a particular CPU/architecture (probably arm has a differnet profile than x86)

[hassila]

Thanks, we'll have a look.