bug: memory leak

[MatthewPattell]

Hi. After a long period of use, we discovered a memory leak. This happens over time. I suspect that the server is accumulating information.

The graph shows how the server crashes every 2-3 months due to lack of memory.

[lega911]

Hi!

There is a few use cases which ijson accumulates (holds) transferred data:

1. if worker in mode "type=get", so ijson holds data up to worker fetches it (cleaned by client's disconnect).
2. if request is called as async or pubsub, then ijson holds it up to worker fetches it.
3. if worker_id is specified: then ijson waits when such worker fetches data.

Also it can be if worker connected in keep-alive mode, but stops fetching data or if it is down but doesn't close a connection, so the connection is considered as active (e.g. bad network among servers), but not sure - just a theory.

What can be checked:

1. For "async" or "worker_id", a number of waiting clients (incoming queue) can grow, possible to see via: curl localhost:8001/ and curl localhost:8001/rpc/details, if number for "workers"/"clients" is to high - means worker/client doesn't fetch data.
2. if to start a debug build (app ijson.debug / ijson_debug), then command "debug" is available (curl localhost:8001/debug) which displays memory-counter, and if it grows while memory consumption, then it's not memory leak, but accumulation of data (data is not fetched by workers).

For now it's not clear if it's leaking and where to start with, I need some use-case how to reproduce a leaking. For my servers ijson has been working for years without restart, no leaking found.

[MatthewPattell]

Thanks for reply. I'm not using 1 or 3 point. My workers healthy and curl localhost:8001/rpc/details doesn't show something strange.

4.2$ sudo curl 127.0.0.1:8001/rpc/details
{"$info":{"version":"0.4.10","uptime":9143104,"connect":221832,"request":437455,"call":216031,"result":215623,"recv":1212719528,"send":1119142635,"io_event":1105495},
"ms/conf":{"last_worker":1703775050,"worker_ids":["2fd81","38604","17080","5ea3e","4f038"],"workers":5,"clients":0}}

everything looks like there are no requests in the queue. Well, I'll keep an eye on it and come back with more details.