queryparser icon indicating copy to clipboard operation
queryparser copied to clipboard

how to extract src/target map of column lineage?

Open tooptoop4 opened this issue 3 years ago • 3 comments

ie srctableA.srcColumn1 = trgtableB.trgColumnX srctableA.srcColumn3 = trgtableB.trgColumnY

tooptoop4 avatar Jul 15 '21 09:07 tooptoop4

+1. Could we get the column lineage info?

wangqinghuan avatar Aug 24 '21 08:08 wangqinghuan

Column lineage, if i remember correctly, isn't currently implemented, and all the original authors (myself included) are not active on this project. However, if you're interested in contributing, I can help approve a PR.

The challenge we had with column lineage is with representing multiple column inputs. some, like SELECT table.username or SELECT avg(table.leftvalue, table.rightvalue) are straightforward, but some are more unintuitive, like SELECT CASE WHEN table.type = 'foo' THEN table.foovalue ELSE table.barvalue END. Representing those in the same output as the rest was not immediately obvious without an overly verbose solution

h4v0kh3l1 avatar Aug 24 '21 20:08 h4v0kh3l1

@wangqinghuan see https://github.com/tooptoop4/presto_sql_lineage

tooptoop4 avatar Sep 02 '21 15:09 tooptoop4