RQ Scheduler very sensitive to restarts / crashes

[Ardiea]

Issue Summary

Environment: Docker / Linux Browser: N/A

If the Scheduler container does not stop cleanly, frequently it will be unable to restart all together. While I don't have details on explicitly creating this condition, it does seem like it is resolved in the upstream RQ scheduler build with version 0.10.0 which enables RQ to run multiple instances at the same time: https://github.com/rq/rq-scheduler/pull/212

scheduler_1         | REDASH[2022-07-18 19:38:00,665][PID:1][INFO][rq_scheduler.scheduler] Registering birth
scheduler_1         | Traceback (most recent call last):
scheduler_1         |   File "/app/manage.py", line 9, in <module>
scheduler_1         |     manager()
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
scheduler_1         |     return self.main(*args, **kwargs)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 586, in main
scheduler_1         |     return super(FlaskGroup, self).main(*args, **kwargs)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
scheduler_1         |     rv = self.invoke(ctx)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
scheduler_1         |     return _process_result(sub_ctx.command.invoke(sub_ctx))
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
scheduler_1         |     return _process_result(sub_ctx.command.invoke(sub_ctx))
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
scheduler_1         |     return ctx.invoke(self.callback, **ctx.params)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
scheduler_1         |     return callback(*args, **kwargs)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
scheduler_1         |     return f(get_current_context(), *args, **kwargs)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
scheduler_1         |     return __ctx.invoke(f, *args, **kwargs)
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
scheduler_1         |     return callback(*args, **kwargs)
scheduler_1         |   File "/app/redash/cli/rq.py", line 31, in scheduler
scheduler_1         |     rq_scheduler.run()
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/rq_scheduler/scheduler.py", line 404, in run
scheduler_1         |     self.register_birth()
scheduler_1         |   File "/usr/local/lib/python3.7/site-packages/rq_scheduler/scheduler.py", line 46, in register_birth
scheduler_1         |     raise ValueError("There's already an active RQ scheduler")
scheduler_1         | ValueError: There's already an active RQ scheduler
scheduler_1         | Sentry is attempting to send 0 pending error messages
scheduler_1         | Waiting up to 2 seconds
scheduler_1         | Press Ctrl-C to quit

Steps to Reproduce

1. N/A, see comments above.

Technical details:

• Redash Version: redash/redash:10.1.0.b50633
• Browser/OS: Linux
• How did you install Redash: Docker

Resolution

Probably just updating requirements.txt to 0.10.0 for rq-scheduler

[susodapop]

Thanks for this report. Have you tested updating to rq-scheduler==0.10.0? Does it improve stability?

[Ardiea]

Yes I have opened the linked PR. Tested and deployed to our QA environments today. Everything looks good so far.

[susodapop]

Thanks. Will review your PR and go from there. This looks to be a solid change 👌