dbt-utils icon indicating copy to clipboard operation
dbt-utils copied to clipboard

feat: add callable where clause in union_relations

Open tnightengale opened this issue 9 months ago • 2 comments

resolves #

This is a:

  • [ ] documentation update
  • [ ] bug fix with no breaking changes
  • [x] new functionality
  • [ ] a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

I noticed that the where parameter in the union_relations macro does not allow one to create unique where clauses for each relation being unioned. In the case of an incremental model, it would be very helpful to be able to configure an incremental where clause for each model separately.

In the simplest form, a project could implement a macro like the following:

{% macro where_clause(relation) %}

{% if execute %}
{% set where_clause %}
{% if is_incremental() %}
occurred_at >= coalesce((select max(occurred_at) from {{ relation }}), '1900-01-01')
{% else %}
    true
{% endif %}
{% endset %}
{% endif %}

{{ where_clause }}

{% endmacro %}

And the pass it to the `union_relations macro like so:

{{
    config(
        materialized="incremental"
    )
}}


{{ dbt_utils.union_relations(
    relations=[
        ref("model_a"),
        ref("model_b")
    ],
    where=where_clause
) }}

Checklist

  • [ ] This code is associated with an Issue which has been triaged and accepted for development.
  • [x] I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • [x] BigQuery
    • [ ] Postgres
    • [ ] Redshift
    • [ ] Snowflake
  • [x] I followed guidelines to ensure that my changes will work on "non-core" adapters by:
    • [x] dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
    • [x] using the limit_zero() macro in place of the literal string: limit 0
    • [x] using dbt.type_* macros instead of explicit datatypes (e.g. dbt.type_timestamp() instead of TIMESTAMP
  • [ ] I have updated the README.md (if applicable)
  • [ ] I have added tests & descriptions to my models (and macros if applicable)
  • [ ] I have added an entry to CHANGELOG.md

tnightengale avatar May 20 '24 03:05 tnightengale