grpc
Potential memory leak related to grpc health probe
Hey folks I have a very simple grpc health endpoint (https://github.com/grpc-ecosystem/grpc-health-probe) and can observe a memory leak coinciding with the regular health check probes using up roughly half a GB every twelve hours in my specific deployment setting.
In fact I could reproduce it with something along the lines of
while true; do sleep 0.001; grpc_health_probe -addr localhost:50051; done
where I can reproduce the memory leak very quickly.
The solution I have found was to tune the grpc options and to add
'grpc.max_connection_idle_ms': 10000, // Close idle connections after 10s
'grpc.max_connection_age_ms': 30000, // Force connection close after 30s
I wanted to raise this here since I have seen multiple folks run into memory leaks and it was hard to find a workaround.
See
- https://github.com/grpc/grpc-node/issues/2068
- https://github.com/grpc-ecosystem/grpc-health-probe/issues/34
Env: Linux, Node v22, @grpc/[email protected]
I believe a memory leak here is unfortunate even if it turns out it's the client disconnecting in wrong ways. Hope this helps.
I think that the max connection idle and max connection age settings that you mention are probably going to be the best solution to this problem. The most likely problem here is that the server doesn't know that a connection is closed, so it will keep tracking the information it has about it. Those settings allow the server to discard connections that haven't been closed.
@murgatroid99 This happens when using the native k8s grpc health probe. I didn't encounter any problems with that for either the java and go grpc sdk so I strongly suspect it's an issue with the node grpc sdk itself.
