apache
Add support for user defined `_retryable_error` in databricks provider
Description
we can let users pass their own _retryable_error method, which will validate if retry is allowed or not.
As of now, databricks operators only accept retry_limit and retry_delay. Adding support for custom retry function and after func should be good enough to handle all custom requirements.
https://github.com/apache/airflow/blob/6b090b31038e6dd86f6efd49d81cf34b73d1e13e/providers/src/airflow/providers/databricks/hooks/databricks_base.py#L127
Use case/motivation
It will allow user to set their own retry logics and also not require change to the provider everytime a user wants to retry for a particular type of exception.
Related issues
https://github.com/apache/airflow/issues/43080
Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
yeah, IMO another option is that we could accept a user supplied list of retryable errors & add those to our _retryable_error validation method. Probably we could also wait to work on this until there is a good demand for this? And users say how they would like it to be exposed as (how much control they would like)?
yeah, IMO another option is that we could accept a user supplied list of retryable errors & add those to our _retryable_error validation method. Probably we could also wait to work on this until there is a good demand for this? And users say how they would like it to be exposed as (how much control they would like)?
I won't start working on this right away. But, if I see any users requesting this, I'll raise a PR
IMO I think it will be a simpler interface if we pass a list of errors to add instead of a function. In particular since rewriting your own function more or less implies to go check the current implementation to make that you have not forgot any special use case.
Yes. list of errors look good and we implemented it elsewhere already (i am quite sure I saw it in other operators). Having a method to pass, but if you implement the operator in the way that it has a public "should_retry(response)" method, then you should be able to implement your custom operator deriving from the base one and rewrite only that method - which is the way the operators should be extended with code.
