velox
velox copied to clipboard
facebookincubator
GCS Set a default policy to retry failures
GCS: Add properties to configure retry policy
GCS may encounter recoverable errors, for example, an authentification error. By default GCS keeps retrying for up to 15 minutes, but this default may be excessive, giving the impression that Velox has become unresponsive. This behaviour can be configured, with options to configure the time to keep retrying or the number of times to retry. However, the GCS connector does not allow to configure neither the retry time nor the retry count.
This change introduces two new properties:
- hive.gcs.max-retry-count: integer The maximum retry counter of transient errors.
- hive.gcs.max-retry-time: integer The maximum time allowed (seconds) to retry transient errors.
Fixes #9264
Deploy Preview for meta-velox canceled.
| Name | Link |
|---|---|
| Latest commit | 207b5f23ac4e12561ece8b528717afde71a128d4 |
| Latest deploy log | https://app.netlify.com/sites/meta-velox/deploys/663171804a6569000879ba2b |
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hello @kgpai and @majetideepak. We recently identified the GCS connector does not allow to configure the retry policy. This PR adds a couple of properties to configure the retry policy.
@tigrux some comments. Can we add a unit test for these configs?
I had to test manually, I do not know how to trigger a failure from the simulator.
@majetideepak @kgpai I addressed most of the feedback, however, I was unable to add UTs (I could not get the simulator to trigger retries) so I added a note stating that I tested manually.