Retry PostgreSQL connection on start-up

[ocharles]

• [x] I have checked the existing issues to avoid duplicates
• [x] I have redacted any info hashes and content metadata from any logs or screenshots attached to this issue

Is your feature request related to a problem? Please describe

I run BitMagnet as a Nomad job, with all tasks in the same group. Nomad doesn't really give a way to add dependencies between tasks, which means my PostgreSQL server process and BitMagnet launch at the same time. BitMagnet launches faster than PostgreSQL, instantly fails, and this causes PostgreSQL to be killed too. Essentially there's a race condition on startup.

Describe the solution you'd like

BitMagnet retries the database connection on start-up, giving PostgreSQL enough time to launch.

Describe alternatives you've considered

I could split my PG server into a separate Nomad group, but this complicates networking (as PG might now run on a completely separate machine).

[ocharles]

Ah, actually I think https://developer.hashicorp.com/nomad/docs/job-specification/lifecycle is what I need (prestart & sidecar). Still, a few retries feels like good practice!

[mgdigital]

I agree we need a grace period with retries on startup...