temporalio
Application Error with a Suggested Retry Interval Duration
Is your feature request related to a problem? Please describe. Given an activity that calls a third-party service to perform a rate limited action, we return an application error when encountering a rate-limit response. Sometimes, the third-party rate limit suggestion is to NOT make another call for X-period of time which is dynamic (e.g. 5s to 90s).
Describe the solution you'd like
It would be nice if the application error could also hold a retry interval value so that the Temporal server would not schedule the activity retry until such time has elapse.
Describe alternatives you've considered
1- Set the activity's RetryPolicy.InitialInterval to the max X-period expected. This is bad because the activity is waiting too long before retrying.
or
2- Use a non-retryable application error and add details to it to then in the workflow make a timer and then retry the call. This bad for many reasons (not listed here because they are probably obvious).
or
3- Don't return an error. Have the activity handle the retrying by waiting and then trying again. This is bad because we're reinventing a feature that's a first-class citizen in Temporal. Additionally, it will take up an active activity slot.
PR to allow workflow tasks to specify a retry delay interval - https://github.com/temporalio/temporal/pull/5946 We still need to update the SDK to send this option before we can make this feature complete. Working on that.
