store: retry bucket store initial sync upon failure

[shaoyi1997]

Signed-off-by: shaoyi1997 [email protected]

This PR fixes #4810.

• [x] I added CHANGELOG entry for this change.
• [ ] Change is not relevant to the end user.

Changes

Currently, initial sync of the bucket store fails without retrying upon a temporary bucket error. This PR wraps bs.initialSync in runutil.RetryWithLog and introduces 2 flags, --initial-sync-retry-interval and --initial-sync-retry-timeout, to allow configuration of the interval and timeout durations for retrying the initial sync.

Verification

Immediately after starting the minio server, start store (gist, adapted from quickstart.sh) Store should retry sync and succeed with initial sync

[matej-g]

@bwplotka those are good points. Why I primarily opened the issue was that the initial sync sometimes fails on the first try even if config is correct, because there is a transient error. But there should be balance between too many retries with bad config vs. transient errors. I'd agree with having some sane defaults for now (e.g. retry max 3x times and then fail initial sync).

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.