Broken Stored Procedure lineage bug

[KsM00000]

Affected module UI

Describe the bug I have an issue with a lineage displaying in the stored procedure entity. So I've build a nice lineage with a script by parsing a stored procedure code. The script added the procedure in the lineage details in the edge between two tables. At first everything was fine, but then suddenly the target table disappeared from the lineage. BUT ONLY in the lineage of the stored procedure entity.

If I look at the lineage in the target table

or any source table, there is no problem.

Now, it's not the only one lineage of the stored procedure, but only one affected. After some digging, I think the problem started after I manually (via the UI) added the column details to the existing edge. Again, everything appeared fine at first, but only disappeared after a while.

I don't use ingestion for the stored procedures.

I tried ES re-inexing (with recreating on). I tried creating lineage second time. Sometimes lineage appears and after some time it disappears (it really does seem to happen randomly). Last time it appeared, but without any column lineage. If I call API 'get lineage by ID' to the stored procedure it returns perfectly fine lineage edge with columns details, including manually added details.

To Reproduce

1. Create lineage between two tables with stored procedure in lineage details using Python SDK (AddLineageRequest).
2. With UI manually try to add links between columns in the existing lineage edge.
3. Wait some time, I guess.

Expected behavior Correct lineage in the stored procedure entity.

Version:

• OS: Ubuntu
• Python version: 3.9
• OpenMetadata version: 1.5.4

[KsM00000]

New developments:

• I was playing around with creating lineages to stored procedures with Python SDK. At some point, I added a lineage between two tables with procedure B in details. But those tables already had a connection with the procedure A (the procedure described in the issue) in the details. So, essentially, I deleted a bunch of lineages (but not all) in the procedure A and added lineages in the procedure B (btw its a bummer we can't have lineages between two tables with different procedures in the details). Aaand after some time I noticed that the lineage in the procedure A was displayed fully again with the target table (excluding deleted lineages between target table and some source tables). After I re-created lineages in full for the procedure A, target table was gone again.

• I created a new lineage in the another procedure via script with Python SDK. I can say for sure that nobody edited this lineage automatically or manually when soon after the target table disappeared. Just like above, lineages were fine for tables and only had displaying troubles in the procedure. Target table appeared again after some time.

I officially have no idea why this happening :/ Maybe it's problems for specific source tables? But why is target table affected? Or maybe there is some kind of timeout... I just don't know..

[harshach]

@karanh37 can we check if its actually bug otherwise close it out

[ulixius9]

fixed by https://github.com/open-metadata/OpenMetadata/pull/19295, available in 1.6.2