More natural sorting of DAG runs in the grid view

[hterik]

Apache Airflow version

2.3.2

What happened

Dag with schedule to run once every hour. Dag was started manually at 12:44, lets call this run 1 At 13:00 the scheduled run started, lets call this run 2. It appears before run 1 in the grid view.

See attached screenshot

What you think should happen instead

Dags in grid view should appear in the order they are started.

How to reproduce

No response

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow==2.3.2 apache-airflow-client==2.1.0 apache-airflow-providers-celery==3.0.0 apache-airflow-providers-cncf-kubernetes==4.0.2 apache-airflow-providers-docker==3.0.0 apache-airflow-providers-ftp==2.1.2 apache-airflow-providers-http==2.1.2 apache-airflow-providers-imap==2.2.3 apache-airflow-providers-postgres==5.0.0 apache-airflow-providers-sqlite==2.1.3

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[bbovenzi]

We currently order the grid by execution_date, but that doesn't always line up with the start date nor data_interval_start.

@uranusjr Should we still use execution_date or should we change that to data_interval_start?

[uranusjr]

data_interval_start is probably better than execution_date, but from what I read the issue, that would still be wrong and we should order by start_date instead (!)

This really depends what we think the runs should be ordered and I can easily see there are different answers for different people (or even the same person under different circumstances), so I guess data_interval_start is as good as any.

[bbovenzi]

The "Run" in the details section of the grid view defaults to data_interval_start and then uses execution_date as a backup. So I guess we should use that for the ordering here and even the tooltip. But, happy to hear a competing suggestion.

[eladkal]

Can we solve both by allowing users to choose the order layout? However I think start_date makes most sense in this view as its simple to understand. If using execution_date/ data_interval and mixing scheduled with manual runs it may be more complecated to understand?

[uranusjr]

I think the data interval would be OK for mixed scheduled and manual runs since AIP-39 took extra care to align them across run types. execution_date would definitely be confusing (because it has different semantics between those types) but is probably OK as a backup. I’m cautious to using start_date since when the run starts doesn’t necessarily contain any logic, but perhaps data_interval_end could be better than _start? Not sure.

[potiuk]

I think either data_interval_start and data_interval_end are good.

Traditionally end woudl have been better but with the new "PlainCronTimetable" we are finally getting into the realm of "normal cron behaviour" (which opens up Airflow to number of different "logical" cases) and with it start seems much more natural.

Somehow I have a feeling that we should be able to choose (per DAG) whether start or end is used. Maybe that is the right approach? We could even attempt to detect which one is "default" for each dag based on timetable.

(Though as usual more options means more complexity).

[uranusjr]

choose (per DAG) whether start or end is used

This is likely too much burden on the user; I imagine most users don’t actually have a good grasp on how the runs should be ordered and would be hard-pressed to choose a “correct” value.

But I wonder if making the logic per-timetable make sense, i.e. for some timetables the start makes more sense, and some the end, so a hook on the timetable class to return a “logical” date (pardon the name) to use in sorting algorithms seems like a reasonable approach.

[potiuk]

But I wonder if making the logic per-timetable make sense, i.e. for some timetables the start makes more sense, and some the end, so a hook on the timetable class to return a “logical” date (pardon the name) to use in sorting algorithms seems like a reasonable approach.

Yep. That would make perfect sense.

[uranusjr]

Still need the UI implementation