br backup/restore retry is too long when there is S3 outage

[fubinzh]

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

• Run br restore from S3
• While restore is in progress, simulate S3 outage
• Check for how long br will retry before it fails

2. What did you expect to see?

• Retry should not be too long, 1 - 3 minutes of retry should be enough per PM's suggestion.

3. What did you see instead?

• Currently br might keep retrying for about 8-9 miniutes.

4. What version of BR and TiDB/TiKV/PD are you using?

br: v5.1.0

5. Operation logs

   • Please upload br.log for BR if possible
   • Please upload tidb-lightning.log for TiDB-Lightning if possible
   • Please upload tikv-importer.log from TiKV-Importer if possible
   • Other interesting logs

6. Configuration of the cluster and the task

   • tidb-lightning.toml for TiDB-Lightning if possible
   • tikv-importer.toml for TiKV-Importer if possible
   • topology.yml if deployed by TiUP

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible