dbt-project-evaluator icon indicating copy to clipboard operation
dbt-project-evaluator copied to clipboard

Avoid `dbt run` failing if the seed has not been loaded

Open b-per opened this issue 2 years ago • 1 comments

Describe the feature

A few people reported issues when they run the project without the seed being executed. A dbt seed or dbt build works (and this is what we have described in the README) but might not be super easy to catch at first.

Instead of having a "default seed" that needs to be run, I was wondering if we could remove it and use get_relation to check if the seed exists in the warehouse or not. And if it doesn't exist we just don't filter out any of the fct tables.

Any opinion from anyone?

b-per avatar Oct 24 '22 09:10 b-per

I am open to it! I don't think it's too much to expect folks to realize that the seed is necessary to execute, but i can also see the argument to make the package as ergonomic as possible -- seems simple enough!

dave-connors-3 avatar Oct 24 '22 13:10 dave-connors-3

What happens if you once had an exception seed (so the seed exists in the warehouse), but you then later delete the seed file from your dbt project?

graciegoheen avatar Jan 05 '23 19:01 graciegoheen

I think we'd want to do something like...

    {% set seed_exists = ref('dbt_project_evaluator_exceptions') is not none %}

    {% if seed_exists %}

And then only apply to filter if the seed exists, but that gives me a compilation error. Model 'model.dbt_project_evaluator.fct_staging_dependent_on_marts_or_intermediate' (models/marts/dag/fct_staging_dependent_on_marts_or_intermediate.sql) depends on a node named 'dbt_project_evaluator_exceptions' which was not found

graciegoheen avatar Jan 05 '23 19:01 graciegoheen

Alternatively, we could do something like:

    {% set seed_exists = load_relation(ref('dbt_project_evaluator_exceptions')) is not none %}

But we'd still need the blank seed in the package, otherwise we'd get the same compilation error.

graciegoheen avatar Jan 05 '23 20:01 graciegoheen