apache
Filter datasets graph by dag_id
Read in all the dag_ids from the dataset dependencies endpoint and allow filtering the datasets graph by dag_id A user can filter via the multi-select or from a tooltip when clicking on a Dag
Also fixed a bug where we didn't always create the sub graphs correctly.
This only filters the graph, not the datasets list for now. It would be ideal to do this logic on the backend (see https://github.com/apache/airflow/issues/37423) to better couple them but this gets us some functionality.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@bbovenzi how are we representing the All and Any relationship here? I suggest we have a representation for All and Any relationship for the dataset.
That PR isn't merged yet. I will work on that shortly though
Looks correct to me functionality wise after testing this.
Only thought I had is Let's say DAG1 produces Dataset1, DAG2 produces Dataset2, DAG3 depends on both Dataset1 and Dataset, and if I filter by DAG1, it shows that DAG3 depends on Dataset1, but it does not give a hint that it may depend on other datasets too. If I click on Filter by DAG on DAG3 icon, it does collect and show all the depending datasets. However, was just thinking if we might want to show a hint that when a particular DAG is filtered and it schedules a DAG, the scheduled DAG may also depend on other datasets?
Actually, we will return the whole dataset "graph" that the dag is connected to. I added an info tooltip to help explain.