Duplicate columns for dbt and AWS Glue siblings

[JustasCe]

Describe the bug Reporting here from associated Slack thread for anyone that might run into the same issue.

Ingesting dbt and Glue seems to produce duplicate columns. With 1 set of columns having no descriptions (coming from Glue) and another set having descriptions (coming from dbt). Screenshot of the issue:

Based on the graphql response and the html generated it seems that Glue uses V2 FieldPath encoding, while dbt uses V1.

Not sure if that's the issue, but searching through the code I couldn't find anything that would do merging on the frontend so guessing this is the issue.

Should we upgrade dbt to V2 or attempt to merge on the UI?

Expected behavior Expected the columns to be merged

Desktop (please complete the following information):

• OS: iOS
• Browser Chrome
• Version 0.10.0

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.

[hsheth2]

Slack thread discussing this issue: https://datahubspace.slack.com/archives/C029A3M079U/p1707163080711289?thread_ts=1679591789.985179&cid=C029A3M079U