Memory consumption when stream reaches limit

[needsaholiday]

What version were you using?

2.10.2 tested with 2.9.22 as well

What environment was the server running in?

Debian 10 x64 Kubernetes / k3s

Let me know if you want some nats-box outputs :)

Is this defect reproducible?

Start nats with jetstream. create 100 streams with a limit of 1MB and filestorage. Publish data until limit is reached. No consumers for the streams exist.

Given the capability you are leveraging, describe your expectation?

I would expect a much lower memory footprint event if the stream "overflows". At best a stable memory footprint as stream is written to disk.

Given the expectation, what is the defect you are observing?

Memory seems to spike as soon as stream limits are reached. I would have expected more disk pressure, but not 10x more memory.

It sits at ~100MB during ramp up. Once it hits the limit, it spikes to ~1GB and settles at ~1.1GB

Disk IO & Disk IO Time & Disk Throughput is stable during the testing period.

Memory does not "recover" after a few hours.

Another question: THe message rate chart is misleading as messages are still coming in, but the total messages in the stream do not change. Do you know a metric that is better suited to track ingest?

[tvojacek]

Does it behave the same if GOMEMLIMIT is applied?

[needsaholiday]

hi @TomasVojacek, Will give it a try later today and get back with the results

[neilalexander]

If possible, please enable the profiling port and capture a memory profile when this happens. https://docs.nats.io/running-a-nats-service/nats_admin/profiling

Then let me know what your preferred method is for sending the profile across, i.e. via NATS Slack, email etc.

[derekcollison]

Any updates on this? This occur with latest 2.10.7?