zstd decoder: memory accumulation and potential data loss in history.ensureBlock()

[iblancasa]

When the buffer needs to grow (capacity < target), the current implementation discards existing data:

  if cap(h.b) < h.allocFrameBuffer {
      h.b = make([]byte, 0, h.allocFrameBuffer)  // ❌ Existing data lost!
      return
  }

Any data in h.b is lost when the buffer is reallocated.

Also, the buffers never shrink. Once a decoder allocates a large buffer (e.g., 8MB for a large frame), it retains that memory permanently, even when subsequent frames are much smaller (e.g., 1MB).

Long-running services that process varying frame sizes will accumulate and hold onto peak memory usage indefinitely.

[klauspost]

Why does this feel like an AI generated issue/pr?

Before I spend more time on this, how is it an issue that doesn't just exist as some AI hallucination?

Buffers don't shrink by design.

[iblancasa]

Hi @klauspost. Thanks for the quick response. Sorry it looks as AI-generated.

I have been trying to improve the memory usage of one OpenTelemetry Collector component and my analysis leaded me to this library.

I saw some memory spikes up to 700MB (at least on my test environment) in the collector when handling zstd-compressed responses. With pprof and some testing, I found the buffers were growing but never shrinking when allocFrameBuffer decreases between frames -which was contributing to the memory raise-. Didn't know it was by design and wanted to contribute an "enhancement".

While doing some testing, I found the data loss thing. Maybe that is ok too.

My bad for focusing on the fix without confirming if this matches your design or something. Thanks again for the feedback—I'll be more careful about context going forward.

Is there anything that can be done in those cases where allocFrameBuffer decreases between frames? Because our current problem is that one backend replies with a big zstd response, we allocate a big buffer and we never get another big response again.

I will close my PR and edit the issue title.