grpc
grpc copied to clipboard
nfultz
too many open files
@nfultz, did you see any issues with letting the service run for a longer time?
Everything seems to work perfectly fine while running a grpc service in Docker for a while, and the service responds to requests, but after ~25 mins the process fails with:
{"log":"E0106 00:37:51.980090342 1 tcp_server_posix.c:245] Failed accept4: Too many open files\n","stream":"stderr","time":"2018-01-06T00:37:51.98024391Z"}
Any chance that the server does not free up some resources in the loops or any other ideas?
I think this is an upstream issue - https://github.com/grpc/grpc/issues/11015
Are you setting up the docker like inst/docker-dev or inst/docker-stable ? You might try the other way.
Good point! I'm using the pretty outdated debian version, will check with a newer version and let you know what I've found.
Also unclear at this point if the grpc cleanup code path is getting executed correctly - do you get this error only when the service is under load or does it do error out even when it's idle?
It's been failing after ~20-25 mins of the last request, so while it's idle -- on the other hand, I've not tried the service for a longer time period with decent load. So far I've just started a dozen of Docker containers and tested with a few queries and then decided to leave those for the weekend -- when I found out about this issue. I will keep you posted if I find something useful (eg on grpc version).
Sorry, did not have the time to look into this in more details -- meanwhile, I'm just restarting the containers every 15 mins as a workaround.