flyte
flyte copied to clipboard
[BUG] [flyte-scheduler] Error handling can lead to very high backoff
Describe the bug
Flyte Scheduler uses backoffs to try and recover from errors in communicating with FlyteAdmin.
But, the backoff logic can lead to extremely large back-offs. https://github.com/flyteorg/flyteadmin/blob/f0e4c890ee4e872d32541a6e9f02adbeb2656f47/scheduler/executor/executor_impl.go#L99
Expected behavior
The back-off should be capped at a few miinutes and lower bound should be slightly higher not 3000ns.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
- [X] Yes
Have you read the Code of Conduct?
- [X] Yes