rainbow-saddle
rainbow-saddle copied to clipboard
SIGHUP not received
Hey,
have you ever had the case where the rainbow-saddle process is up. The gunicorn arbiter and workers are up. The pid file is as expected
BUT sending a HUP to rainbow-saddle does absolutely nothing (no log trace, no nothing)?
I don't know what to do or how to investigate short of adding traces in rainbow-saddle code (and still I'm not sure how to go about the issue)
To be clear, I can't reproduce it at will, it just happens from time to time on our server and I need to do a manual restart (getting the server out of the load balancer and stuff like that first)
Hey,
so I found the issue.
Actually rainbow-saddle assumes strongly that the gunicorn second arbiter will start and produce the .pid.2
file. If it does not, for whatever reason
- rainbow-saddle is stuck in an infinite loop waiting for the file.
- gunicorn still runs the old arbiter
What is very problematic here is that it's all silent. The rainbow-saddle process is still up, gunicorn is still up too. The only thing is a stack trace in the gunicorn error log, but since the old master still runs, it actually gets lost in the uninterrupted activity