sqllineage
sqllineage copied to clipboard
Extractors throwing away column metadata?
I am hoping to use this tool to get lineage from dbt models. What I really want is column lineage and it seems that this tool collects that..but then throws it away?
I'm looking at the below from here
statement_segment = statement_segments[0]
for extractor in [
extractor_cls(self._dialect, metadata_provider)
for extractor_cls in BaseExtractor.__subclasses__()
]:
if extractor.can_extract(statement_segment.type):
lineage_holder = extractor.extract(
statement_segment, AnalyzerContext()
)
return StatementLineageHolder.of(lineage_holder)
The extractors are initialized in the list comprehension and extractor.extract
is called if they are able to extract data. At the end of this step the extractor's columns
attribute has a list of column objects from the segments. But at the end of this the column data is thrown away with the extractor and when I later run lr.get_column_lineage()
I get back an empty list.
Is this by design? I have been trying to read through the docs and issues over the last hour or so to determine what the goals/functionality of the column lineage are but I'm having a hard time figuring that out.