SolidQueue stops processing any jobs, but does not actually exit, when DB restarts unless --skip-recurring is set

[elondres-mim]

Steps to reproduce:

• Have some jobs defined, including at least one recurring job.
• Run solidqueue
• Restart the database (for example on a mac, brew services restart mariadb, or on linux, systemctl restart mariadb)
• Notice that an error appears in the terminal that ran solidqueue, but it doesn't exit (not returned to shell), and no jobs are processed in the future

If you do this again but run with --skip-recurring, solidqueue will print the error and then exit. This is useful because when running solidqueue in a process monitor such as systemd, the solidqueue process exiting will cause the monitor to restart it - if the database restarted just once, this means no future jobs are missed.

[elondres-mim]

I am also interested in helping fix this issue if that would be desired; is there a SolidQueue contributing guide somewhere?

[p-schlickmann]

Hi @elondres-mim, thank you for your interest in improving Solid Queue!

I'm afraid there is no contributing guide, yet.

I have confirmed this issue happens when running Solid Queue as a separate proccess (not as a Puma plugin).

Here's what happens after MySQL is restarted:

If you try to enqueue a normal job, it also fails (JobResult count stays 25):

I have rails 8.1.1, solid_queue 1.2.4 and MySQL 8.0.31

[p-schlickmann]

Actually, I have just confirmed this also happens when running SolidQueue as a puma plugin.

After MySQL restarts, jobs are enqueued, but never performed.

@rosa should SolidQueue exit or try to recover in this scenario? I'd be happy to submit a fix