rainbow-saddle icon indicating copy to clipboard operation
rainbow-saddle copied to clipboard

SIGHUP not received

Open gbataille opened this issue 7 years ago • 1 comments

Hey,

have you ever had the case where the rainbow-saddle process is up. The gunicorn arbiter and workers are up. The pid file is as expected

BUT sending a HUP to rainbow-saddle does absolutely nothing (no log trace, no nothing)?

I don't know what to do or how to investigate short of adding traces in rainbow-saddle code (and still I'm not sure how to go about the issue)

To be clear, I can't reproduce it at will, it just happens from time to time on our server and I need to do a manual restart (getting the server out of the load balancer and stuff like that first)

gbataille avatar Mar 16 '17 10:03 gbataille

Hey,

so I found the issue. Actually rainbow-saddle assumes strongly that the gunicorn second arbiter will start and produce the .pid.2 file. If it does not, for whatever reason

  • rainbow-saddle is stuck in an infinite loop waiting for the file.
  • gunicorn still runs the old arbiter

What is very problematic here is that it's all silent. The rainbow-saddle process is still up, gunicorn is still up too. The only thing is a stack trace in the gunicorn error log, but since the old master still runs, it actually gets lost in the uninterrupted activity

gbataille avatar Mar 22 '17 07:03 gbataille