Unified dbt/bigquery datasets not shown as lookml upstreams in UI

[honeyp0t]

Describe the bug The dbt ingestion will, by default, create two dataset urns per underlying table -- a dbt dataset and a bigquery dataset. As far as I can tell this is a requirement if you want to be able to see the awesome new dbt validation results view in the datahub ui. However, this seems to break lineage in the ui, at least for lookml.

To Reproduce Steps to reproduce the behavior:

1. Ingest a dbt table that is depended on by a lookml definition. Make sure both bigquery and dbt urns are created.
2. Ingest the corresponding lookml definition

Actual behaviour

• When looking at the lineage of the lookml definition in the datahub ui, there is no upstream shown

Expected behaviour

• When looking at the lineage of the lookml definition in the datahub ui, the upstream is visible

Desktop (please complete the following information):

• OS: MacOS
• Browser: Chrome
• Version: 104.0.5112.101

Additional context

• When getting urn data in the datahub cli, the upstream is shown in the JSON response (datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:looker,mycompany.view.employees,DEV)")
• If you remove the ingested dbt nodes (datahub delete --platform dbt --env DEV --hard) and refresh the datahub ui, the lineage is shown again.

[gabe-lyons]

Hey @honeyp0t! Did you also run bigquery ingestion? You will need to run both bigquery and dbt ingestion in order to see lineage function as expected.

[honeyp0t]

Hi gabe! Sorry for late response. We run dbt ingestion without separate bigquery ingestion, with the dbt ingestion module configured to create nodes for both.

This seems to be working again, with lineage from ingested lookml definitions. We did discover some issues on our end in our multi-env setup, and I think this is the main cause of the issues we experienced.

I'm unable to reproduce using the STR, so I will close this issue.