spark-sql-flow-plugin icon indicating copy to clipboard operation
spark-sql-flow-plugin copied to clipboard

Published 20 hours ago • verified publisher icon maropu

Lineage Data can be directly output in JSON format？

Open melin opened this issue 2 years ago • 3 comments

Lineage Data can be directly output in JSON format？ which is convenient to store in the graph database。

melin avatar Dec 08 '21 05:12 melin

Ah, it might be worth doing it, but is there a common JSON format to represent a directed acyclic graph?

maropu avatar Dec 09 '21 23:12 maropu

The format is altas Lineage JSON: https://atlas.apache.org/api/v2/json_AtlasLineageInfo.html

melin avatar Dec 10 '21 01:12 melin

INFO: I added an interface to define an output format for exporting a generated graph into other systems in https://github.com/maropu/spark-sql-flow-plugin/commit/abb1f0f2f9591d3f511948a57ae8fde0032e25ff: https://github.com/maropu/spark-sql-flow-plugin/blob/abb1f0f2f9591d3f511948a57ae8fde0032e25ff/src/test/scala/org/apache/spark/sql/SQLFlowSuite.scala#L175-L190

maropu avatar Dec 13 '21 08:12 maropu