airflow

airflow copied to clipboard

Closes: #39675

Now, on the home page, the two sets of buttons (filter DAG states and DAG Run states work together). I have had to add a new button on the home page to view all states:

I've had to add a few SQL queries to ensure that the number next to each label is correct. Without them, if a DAG has multiple running DAG Runs, it counts them all. For example: we have 1 DAG with 3 running DAG runs --> without the new queries, a (3) appears, instead of a (1).

Moreover, the numbers next to the label may seem strange at first, since they change continuously according to the applied filters.

I initially thought of making this work as the TAGs filter, but it is nice to see the count next to the label.

@PApostol since you developed this initially: what do you think?

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Good work, just a couple of questions for my understanding:

1. What's "All states" required for? Maybe "Running" and "Failed" can just work with the existing "All", "Active", "Paused"?

2. Regarding the new SQL to count all DAG runs - not sure what the best approach is here. Do we want to see how many unique DAGs are currently running regardless of how many instances of the same DAG are running (current implementation)? Or do we want to see the total number of DAG runs (even if all come from the same DAG, say if you're doing a backloading).

Perhaps some comments from the Airflow maintainers would help here.

P.S. did you mean to link #39675 instead of #30429 by any chance?

Good work, just a couple of questions for my understanding:

1. What's "All states" required for? Maybe "Running" and "Failed" can just work with the existing "All", "Active", "Paused"?

I added this button in order to reset the Running/Failed filter. For example:

• I select "Active"
• I select "Failed"

I now want to see "Active" + "Any state". What should I do? By adding this button, you can do it.

Moreover, you clearly show that the properties filtered by each set is different. One filters Dag properties, while the other focuses on DagRun

2. Regarding the new SQL to count all DAG runs - not sure what the best approach is here. Do we want to see how many unique DAGs are currently running regardless of how many instances of the same DAG are running (current implementation)? Or do we want to see the total number of DAG runs (even if all come from the same DAG, say if you're doing a backloading).

Perhaps some comments from the Airflow maintainers would help here.

While I was developing this I found it very confusing to see a "Running (8)" but only 1 DAG on the list (with 8 running DAG runs). I believe that the # of DAGs running is better displayed on the Cluster Activity page.

P.S. did you mean to link #39675 instead of #30429 by any chance?

You are right! Thanks for seeing this!

I added this button in order to reset the Running/Failed filter. For example:

• I select "Active"
• I select "Failed"

I now want to see "Active" + "Any state". What should I do? By adding this button, you can do it.

Might it be cleaner to just unselect failed? There are only two buttons, so I don't know that it makes sense to include an "all states" button when there are only two states to choose from.

Might it be cleaner to just unselect failed? There are only two buttons, so I don't know that it makes sense to include an "all states" button when there are only two states to choose from.

I liked the idea, so I implemented it. Now it looks like the current version:

Might it be cleaner to just unselect failed? There are only two buttons, so I don't know that it makes sense to include an "all states" button when there are only two states to choose from.

I liked the idea, so I implemented it. Now it looks like the current version:

Might it be cleaner to just unselect failed? There are only two buttons, so I don't know that it makes sense to include an "all states" button when there are only two states to choose from.

I liked the idea, so I implemented it. Now it looks like the current version:

I like this. Let's get rid of the ?lastrun=all_states param, that should be the default when nothing is specified.

@bbovenzi done! Similarly to the tags I added the "reset_filter" param to reset the cookie when we deselect the filter. However, the user does not see it, since it immediately redirects.

I agree that the count for Running and Failed should just be the count of dags and not runs.

I am less sure if we only want to filter by lastrun. I would want to know if a dag is running, if it thats not the most recent run (backfill or manually clearing).

I agree that the count for Running and Failed should just be the count of dags and not runs.

I am less sure if we only want to filter by lastrun. I would want to know if a dag is running, if it thats not the most recent run (backfill or manually clearing).

@bbovenzi Even though the param is called lastrun, the query looks if there is any running instance of the DAG. So backfills should also be incldued. If you feel that another name for the param would be better, let me know and I'll change it!