spline-spark-agent icon indicating copy to clipboard operation
spline-spark-agent copied to clipboard

Whether to consider adding column levels

Open javaht opened this issue 2 years ago • 4 comments

Whether to consider adding column levels ?

javaht avatar Sep 12 '23 06:09 javaht

What do you mean?

wajda avatar Sep 12 '23 09:09 wajda

Column level data lineage

javaht avatar Sep 12 '23 09:09 javaht

Yes, but how exactly would you like it to be represented? What output is missing from the agent?

wajda avatar Sep 12 '23 09:09 wajda

The output JSON contains attributes definitions. Each attribute has a childRef property that refers to either another attribute (the one it depends on and derives the value from), or an expression that was used to calculate the values for the given attribute. This forms the attribute (or column) level lineage.

wajda avatar Sep 12 '23 14:09 wajda