Dex doesn't retry when database not available during start

[kaushikjeet]

Preflight Checklist

• [X] I agree to follow the Code of Conduct that this project adheres to.
• [X] I have searched the issue tracker for an issue that matches the one I want to file, without success.
• [X] I am not looking for support or already pursued the available support channels without success.

Version

2.32.0

Storage Type

Postgres

Installation Type

Binary, Custom container image

Expected Behavior

Dex should retry database connections instead failing to start if database not available. In container world its difficult to align pods startup sequence in distributed environment.

Actual Behavior

Dex failed to start if unable to connect with Postgres during start time. It error out with message "failed to initialize storage:"

Steps To Reproduce

1. configure postgres connector with Dex.
2. start Dex without bringing databases first.
3. Dex will fail to start with error "failed to initialize storage:"

Additional Information

We have applications deployed over K3S, Dex is one of the pod having Postgres database connector. It seems to be mandatory have database available for Dex to start as it try to connect with Postgres and error out. It doesn't have logic to retry connection when failed.

Configuration

No response

Logs

failed to initialize storage:

[nabokihms]

@kaushikjeet Yes, I believe there are many cases when Dex does not start because of external dependencies, such as other providers or databases. This is unwanted behavior, and it would be nice to have this issue fixed.

[kaushikjeet]

@nabokihms , shall we try fix and create PR for this behaviour?

[nabokihms]

Yeah, sure, go ahead. I believe this is a good improvement.

[MoeBensu]

@nabokihms I put something together. PR should be ready for review.