dbt-core icon indicating copy to clipboard operation
dbt-core copied to clipboard

[Bug] state:modified.macros can detect false positives if package macro has same name as builtin (even though it won't be used)

Open jtcohen6 opened this issue 1 month ago • 0 comments

Is this a new bug in dbt-core?

  • [X] I believe this is a new bug in dbt-core
  • [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Installing a package that includes a macro with the same name as a builtin macro can trigger false positives for state:modified.macros — even though that macro will not override the builtin in actuality / at runtime.

Expected Behavior

state:modified.macros reflects changes to macros that are actually being called at runtime, and fully ignore same-named macros in packages that won't actually ever be called.

Steps To Reproduce

Let's say every model in my project depends on a certain macro. For example, that macro could be:

Let's say that macro calls another builtin macro - run_query. (This is the case for dbt_snow_mask.apply_masking_policy, which calls it via apply_masking_policy_list_for_sources or apply_masking_policy_list_for_models.)

Let's say you install a new package that has a custom implementation of run_query:

{% macro run_query(query) %}
    {{ exceptions.raise_compiler_error("you aren't supposed to call me") }}
{% endmacro %}

This has no functional effect on your project, because builtin macros can only be overridden if defined within the root project. Even if the builtin run_query macro were dispatchable (it's not), it wouldn't be actually used until the root project explicitly opts in via dispatch config.

And yet. As soon as you install that package, and run dbt ls -s state:modified, you'll see that every single model is selected as "modified." Why? Because dbt seems to think that the post-hook

with open("target/manifest.json") as f:
     d = json.load(f)

d['macros']['macro.dbt_snow_mask.apply_masking_policy_list_for_sources']['depends_on']
{'macros': ['macro.dbt_snow_mask.get_meta_objects', 'macro.dbt_utils.get_query_results_as_dict', 'macro.localpackage.run_query']}

Why is 'macro.localpackage.run_query' showing up there? At runtime, this will actually be macro.dbt.run_query!

Relevant log output

No response

Environment

- OS: Sonoma 14.4.1
- Python: 3.10.11
- dbt: 1.8.0 / main

Which database adapter are you using with dbt?

No response

Additional Context

No response

jtcohen6 avatar Jun 07 '24 21:06 jtcohen6