sqllineage icon indicating copy to clipboard operation
sqllineage copied to clipboard

Extractors throwing away column metadata?

Open NodeJSmith opened this issue 5 months ago • 0 comments

I am hoping to use this tool to get lineage from dbt models. What I really want is column lineage and it seems that this tool collects that..but then throws it away?

I'm looking at the below from here

            statement_segment = statement_segments[0]
            for extractor in [
                extractor_cls(self._dialect, metadata_provider)
                for extractor_cls in BaseExtractor.__subclasses__()
            ]:
                if extractor.can_extract(statement_segment.type):
                    lineage_holder = extractor.extract(
                        statement_segment, AnalyzerContext()
                    )
                    return StatementLineageHolder.of(lineage_holder)

The extractors are initialized in the list comprehension and extractor.extract is called if they are able to extract data. At the end of this step the extractor's columns attribute has a list of column objects from the segments. But at the end of this the column data is thrown away with the extractor and when I later run lr.get_column_lineage() I get back an empty list.

Is this by design? I have been trying to read through the docs and issues over the last hour or so to determine what the goals/functionality of the column lineage are but I'm having a hard time figuring that out.

NodeJSmith avatar Sep 04 '24 22:09 NodeJSmith