airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Add disable retry flag on backfill

Open domagojrazum opened this issue 3 years ago • 7 comments

Oftentimes while backfilling the data it's known that some day will fail due to some source being unable to provide the data or due to retention period (on the source). Meaning, backfill take longer than it should. For that reason disable_retry is implemented so the backfill set the states of all TIs to FAILED on a first UP_FOR_RETRY show up.

domagojrazum avatar May 20 '22 12:05 domagojrazum

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: [email protected] Slack: https://s.apache.org/airflow-slack

boring-cyborg[bot] avatar May 20 '22 12:05 boring-cyborg[bot]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

github-actions[bot] avatar May 22 '22 11:05 github-actions[bot]

I ihnk this one needs rebase/confict resolving and @ashb re-review afterwards @domagojrazum

potiuk avatar Aug 23 '22 11:08 potiuk

Rebased and conflicts are resolved. @potiuk

domagojrazum avatar Sep 05 '22 12:09 domagojrazum

Rebased to test it all works fine for the smaller set of tests (public runners does not seem to work well when we want to run all k8S tests).

potiuk avatar Sep 20 '22 12:09 potiuk

@potiuk thank you. Looks like checks have passed.

domagojrazum avatar Sep 20 '22 16:09 domagojrazum

@ashb?

potiuk avatar Sep 20 '22 19:09 potiuk

@ashb @potiuk

domagojrazum avatar Oct 10 '22 08:10 domagojrazum

@ashb? I just rebased it, and this one is blocked with your comment (responded to)

potiuk avatar Oct 28 '22 15:10 potiuk

Static checks need fixing then/

potiuk avatar Oct 31 '22 05:10 potiuk

@potiuk fixed

domagojrazum avatar Nov 02 '22 08:11 domagojrazum