[Tableau Ingestion] Lineages for CustomSQLTables are not always correct

[cuong-pham]

Describe the bug For CustomSQLTable, the graphql return the following

{
   "tables": {...}
   "query": {...}
   "columns": {...}
}

Currently, the logic take columns fields and get the embedded datasource associated with the columns. However what i noticed is that the datasource information is not always matchup with the query in the CustomSQLTable. Sometime BI developers only reuse the datasource connection but query different tables instead. For simple query, tables properties has the most accurate information. On the other extreme end, when the query is very complex, both tables and upstreamTables in columns are empty, we have to rely parsing query. I have tried to use SqlLineageSQLParserImpl but the library it uses sqllineage is quite buggy.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.

[hsheth2]

@cuong-pham could you confirm that this is still an issue?