datahub
datahub copied to clipboard
redshift unload lineage cannot output mcp
Describe the bug
There is an issue with Redshift ingestion; when S3 is used as the downstream (include_unload_lineage
), it cannot produce MCP outputs.
Screenshots
There is a bug with self.aggregator._is_temp_table
;
- s3://xxx is not in all_tables
- Incorrect platform so we cannot output the unload lineage mcp.
https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/ingestion/source/redshift/lineage_v2.py#L97-L116
https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/sql_parsing/sql_parsing_aggregator.py#L514-L524
https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/sql_parsing/sql_parsing_aggregator.py#L1112-L1117
Desktop (please complete the following information):
- Version [0.14.1]