vscode-dbt-power-user

vscode-dbt-power-user copied to clipboard

Expected behavior

When using a cross-project reference with dbt-loom, dbt compile should work in dbt-power-user, as it is working when running the cmd in CLI.

Actual behavior

dbt compile in dbt-power-user does not find the model from the other project

Steps To Reproduce

We started from a setup where we had a cross-project dependency (project B) which was defined in the packages.yml We wanted to test the same using dbt-loom as it reduces significantly the parsing time (our project dependency has lots of models)

1. Install dbt-loom
2. remove project B from packages.yml
3. define manifest location of project B in dbt_loom.config.yml
4. dbt clean && dbt deps
5. dbt run

Step 5 fails when done in dbt-power-user by clicking the 'run model' button. However a manual dbt run executed in CLI works, which shows that our dbt-loom setup in general works.

In the Log output below, you can see that there is a little difference. dbt-loom creates two log entries when run in CLI:

• dbt-loom: Loading manifest for data_platform from file
• dbt-loom: Injecting nodes

However in the log of a run from dbt-power-user only the second log entry is printed. I suspect that the manifest loading is skipped for some reason, however I don't know what triggers it normally.

Log output/Screenshots

Log Output when running a model from within dbt-power-user:

Executing task: dbt run --select test --project-dir /data/data-marts --profiles-dir /data/data-marts 15:20:23 Running with dbt=1.7.3 15:20:23 Registered adapter: bigquery=1.7.2 dbt-loom: Injecting nodes 15:20:25 Encountered an error: Compilation Error Model 'model.data_marts.test' (models/test.sql) depends on a node named 'abonnement' in package or project 'data_platform' which was not found

Log Output when running the command in the terminal window in VSCode:

dbt run --select test --project-dir /data/data-marts --profiles-dir /data/data-marts 15:28:21 Running with dbt=1.7.3 dbt-loom: Loading manifest for data_platform from file 15:28:23 Registered adapter: bigquery=1.7.2 dbt-loom: Injecting nodes 15:28:24 Found 2367 models, 0 sources, 0 exposures, 0 metrics, 448 macros, 0 groups, 0 semantic models 15:28:24 15:28:28 Concurrency: 1 threads (target='local') 15:28:28 15:28:28 1 of 1 START sql table model feature_do_000_loom.test ......... [RUN] 15:28:33 1 of 1 OK created sql table model feature_do_000_loom.test .... [CREATE TABLE (3.7m rows, 114.0 MiB processed) in 4.37s] 15:28:33 15:28:33 Finished running 1 table model in 0 hours 0 minutes and 8.66 seconds (8.66s). 15:28:33 15:28:33 Completed successfully 15:28:33 15:28:33 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Operating System

Debian 11

dbt version

1.7.3

dbt Adapter

bigquery

dbt Power User version

0.34.0

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

I looked a bit further into this and was able to make the "run model" working from within dbt-power-user. One issue was to use a relative path in dbt_loom.config.yml and also that this config file was not found when executing a dbt command from dbt-power-user as the working directory is not the one of the dbt project. So I did: a) set the env variable DBT_LOOM_CONFIG_PATH to the full path of the dbt_loom.config.yml b) define the full path to the other projects manifest.json in dbt_loom.config.yml

As a result, running models works now in dbt-power-user, but I still don't see any lineage, or any relations (child/parent) in the dbt panel on the left. It still feels like dbt-power-user cannot compile the project. I couldn't find any error messages in the logs that help me undersand the problem.

@alangner : Can you try running the extension in debug mode and getting more insights on this issue?

@mdesmet I tried to better understand the behavior by using the debug mode. So far I found nothing interesting by looking at extension.js file. How do you debug the python code of the extension? I would like to see what happens inside the project.safe_parse_project() function

Hey @alangner, is this still a requirement? We are working on the roadmap for the next few months and were wondering if this needs to be prioritized.

For us it would be super nice if we could use dbt-loom together with dbt Power User. We are struggling in our current setup as we have a larger „core” dbt project with many models (around 4k). And then we have further dbt projects that want to reference models from the „core” project. Importing it as a package dependency is suboptimal. First, it increases project parsing duration significantly. Second, the integration of these two projects is too tight. For example, a developer of the satellite projects could accidentally run models from the „core” project (if he has the permissions), while the core project is only supposed to provide models and nothing more. (Plus one inherits also packages from the core project which could generate version conflicts etc.)

This problem was actually addressed in dbt Cloud (https://docs.getdbt.com/docs/collaborate/govern/project-dependencies) But these project dependencies are not available in dbt-core. This where dbt-loom comes into play. It works very well for us on the CLI. But most of our developers like to work with the dbt Power User extension :) so we haven’t switched to using dbt-loom (just ran some tests).

If dbt Power User would support dbt-loom, it would bring another feature that is only available in dbt Cloud to user working with dbt-core. I think this is actually what your extension is already doing for many users. Creating an environment with similar features that you otherwise only have in dbt Cloud (of course you provide more unqiue features on top of that as well). I hope this helps you to decide whether this feature should come on the roadmap. Let me know if you need more information about our development setup, or why we would profit so much from this feature.

I tried it again with dbt-loom 0.5.3 and dbt Power User 0.39.12

The lineage view is now working !

What does not work yet is to run a model from within the extension. I get this error: Compilation Error Model 'modelA' depends on a node named 'modelB' in package or project 'refProject' which was not found.

Can you provide repro steps as we are not able to reproduce this issue? On our we can succesfully run models without getting this error.

@alangner: I am closing this ticket as the issue has been confirmed to be resolved now.

Feature is available in version 0.39.12 onwards.