astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Open lineage bug workaround

Open sunank200 opened this issue 1 year ago • 3 comments

Describe the bug Message from @conorbev So while waiting for the release I tried it out from main and I do see the TransformOperator producing Inputs and Outputs now, I also did not need to set the custom extractor env var. It seemed to work fine with the default extractor. Very cool!

(https://astronomer.slack.com/archives/D044NMSCBTR/p1669860239092409) I think I've found the reason why the SQL that the TransformOperator was supplying in the facet did not show up. It's because the SQL facet is only being supplied in the COMPLETE OpenLineage event and you are running into this bug: https://github.com/MarquezProject/marquez/issues/2230 ( I confirmed this by manually copying your SQL facet into the START event that you send and then the SQL does show up as expected: https://cloud.astronomer.io/lineage/graph/job/conor-astrosdk4/calculate_popular_movies.top_five_animations ) #2230 Only first job context is taken into consideration.

So I think we certainly need to fix that bug in Marquez either way, but in the meantime, I wanted to check with you: Is it possible for you to set the facets during the START event or you only know it at the end ?

sunank200 avatar Dec 01 '22 08:12 sunank200