Rene Treffer

The example config is working working right now. However the exported metrics do not reflect the `stats_prefix` in prometheus. Here is a rendering of the relevant stats from this service...

I got nerd sniped by this a bit... I have an _experimental_ [zstd encoder pool](https://github.com/rtreffer/sarama/blob/zstd-pool/zstd_pool.go) (read: don't use it without full understanding of the code). An initial (probably flawed) pseudo-benchmark...

Ok, I managed to get a [benchmark](https://github.com/rtreffer/sarama/blob/zstd-pool/zstd_test.go#L35) going to test different variants of pool configurations that is hopefully somewhat informative 1. GOMAXPROCS (1 & 4) 3. max idle encoders (1,...

I think I finally found an easy way to express this all in code: https://github.com/rtreffer/sarama/commit/23168b5f68474ff129c3fa89d8d12cfdb8ac766a ``` $ git diff --stat HEAD~1 zstd.go | 53 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+),...

I've been pretty busy over the last few days, I expect to have some time for this over the weekend

I am sorry for the wall of text, I'll go over * history of the original PR * the issue described * encoder concurrency * the solution space (IMHO) ##...

So... **In the original PR I wrongly assumed** that we would be tied to GOMAXPROCS as the upper bound for zstd encoders. I just wrote a test case that shows...

@klauspost unfortunately the concurrency setting does not have the desired outcome. The main problem is the greedy allocation in [zstd/encoder.go#L87-L90](https://github.com/klauspost/compress/blob/v1.17.9/zstd/encoder.go#L87-L90) The problem in the past was that on high core...

So 3GB-4GB on 96 cores without setting the concurrency is roughly in line with what I was seeing. Setting the compression level to best cut our bills >50% with no...