Lineages - If we update any existing airflow dags (removing some old tasks) are not removing those task from lineages

[RahulKSinghal]

Steps To reproduce this issue is ..

1.Create a Airflow pipeline dag as below and run it.

• T1 -> T2 -> T3 -> T4 -> T5

We will have the lineages of that dag in datahub as well.

2. Now go back to Airflow dag and do the changes in the dag one moving the T2, T3, T4 in a task group and keep T1 and T5 as is like below.

• T1 -> [ T2, T3, T4] ->T5

3. Now check in the datahub lineages although we can see the new lineages has been generated for T2, T3, T4 as per the task group they has been added, but the older lineages of T2, T3, T4 as per step #1 has not been removed and still persisting.

Ideally the older lineages should get removed.

[swaroopjagadish]

@treff7es is looking into fixing this

[siddiquebagwan]

Hi @RahulKSinghal / @treff7es Is it fixed?

[treff7es]

It is not fixed yet we should figure out some way to support the above

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.