Fixed issue of new dag getting old dataset events.

[tosheer]

If a new dataset triggreed DAG is created for an already existing dataset. (Dataset has already existing dataset events) DAG see all dataset events from very first event for dataset.

Fixes: https://github.com/apache/airflow/issues/39456

[boring-cyborg[bot]]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: [email protected] Slack: https://s.apache.org/airflow-slack

[jedcunningham]

cc @uranusjr

[uranusjr]

To be honest I don’t really consider the current behaviour (in 2.9.1) a bug, but I can see why people expect it this way. I think we probably need an entry in newsfragment to warn users about possible breakages.

[uranusjr]

Also the tests need improvements, as mentioned in above reviews.

[tosheer]

@RNHTTR / @uranusjr Can you guys please review it and let me know if it needs any adjustments.

[RNHTTR]

Looks good to me. Pending @RNHTTR’s comments.

LGTM as well, but I'm unable to mark the comments as resolved :\

(Also the tests need to be addressed)

[tosheer]

@RNHTTR All tests are passing now.

@uranusjr can you merge the PR?

[potiuk]

Tests are failing.

[tosheer]

@potiuk Test are passing now. Can you merged

[Dev-iL]

(Lesson for myself - don't post PR comments on Slack. Anyway, here's a copy of my comment from a few days ago:)

Perhaps it makes sense to have this behaviour user-configurable? Seeing past dataset events sounds useful for backfilling (i.e. when a new process becomes available for old data).

It's not clear from the docs how catchup works for dataset schedules, but I'd imagine that:

• dataset schedule + catchup=True -> current behavior (see all past events]).
• dataset schedule + catchup=False -> the behavior you added (see only new events).

If we take the route suggested above, we should consider what it'd mean for a DatasetOrTimeSchedule + catchup=True (I'd say either forbidden or add all runs from both schedules).

[uranusjr]

What do you think of the comment on still being able to get the old events in some way @tosheer?

[uranusjr]

3.0 is a point when we can break things so let’s consider this before we do that.

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.