dbt-core icon indicating copy to clipboard operation
dbt-core copied to clipboard

[Epic] state:modified should Actually (only) select the modified resources

Open graciegoheen opened this issue 1 year ago • 3 comments

We currently call out a lot of caveats to state comparison in our docs.

There are many scenarios where executing dbt list --select state:modified over or under selects the appropriate resources. This leads to:

  • confusion "why are these models selected as state:modified, I haven't changed them"
  • performance concerns "why am I running 60 models when I only changed 2"
  • potentially bad code sneaking it's way into production "i checked my changes using a Slim CI job, but these models weren't run, now production is broken"

While it's unlikely we can cover ever single edge case here, we should improve state:modified so that it Ideally actually only selects the modified resources.

This epic will track the relevant bugs/enhancements related to state:modified.

### Issues
- [ ] https://github.com/dbt-labs/dbt-core/issues/10002
- [ ] https://github.com/dbt-labs/dbt-core/issues/9548
- [ ] https://github.com/dbt-labs/dbt-core/issues/9563
- [ ] https://github.com/dbt-labs/dbt-core/issues/9564
- [ ] https://github.com/dbt-labs/dbt-core/issues/4304
- [ ] https://github.com/dbt-labs/dbt-core/issues/10138
- [ ] https://github.com/dbt-labs/dbt-core/issues/9573
- [ ] https://github.com/dbt-labs/dbt-core/issues/8727

graciegoheen avatar Feb 13 '24 02:02 graciegoheen

What would be lost by disallowing env-aware logic outside the dbt_project.yml?

matt-winkler avatar Feb 13 '24 21:02 matt-winkler

Lots of examples here: https://discourse.getdbt.com/t/why-are-ci-jobs-state-method-building-more-less-models-nodes-than-expected/7504 An example of env_var logic here , which seems impossible to keep out of state:modified

patkearns10 avatar Feb 14 '24 03:02 patkearns10

For those following along, would love to hear your thoughts over in this discussion https://github.com/dbt-labs/dbt-core/discussions/10518

graciegoheen avatar Aug 02 '24 19:08 graciegoheen