datahub icon indicating copy to clipboard operation
datahub copied to clipboard

redshift unload lineage cannot output mcp

Open yingyingqiqi opened this issue 4 months ago • 0 comments

Describe the bug There is an issue with Redshift ingestion; when S3 is used as the downstream (include_unload_lineage), it cannot produce MCP outputs.

Screenshots There is a bug with self.aggregator._is_temp_table ;

  • s3://xxx is not in all_tables
  • Incorrect platform so we cannot output the unload lineage mcp.

https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/ingestion/source/redshift/lineage_v2.py#L97-L116

https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/sql_parsing/sql_parsing_aggregator.py#L514-L524

https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/sql_parsing/sql_parsing_aggregator.py#L1112-L1117

Desktop (please complete the following information):

  • Version [0.14.1]

yingyingqiqi avatar Oct 10 '24 07:10 yingyingqiqi