dbt-invoke icon indicating copy to clipboard operation
dbt-invoke copied to clipboard

possibility to use templated properties to apply by default on file update

Open david-whimsical opened this issue 1 year ago • 1 comments

hey, first of all, thanks for building this package, it's been a pleasure using it!


Our team recently started to use what's in the meta field, most notably to set the meta.owner property:

version: 2
models:
- name: dim_user
  description: >
    Some description
  meta:
    owner: '@data-team'
  columns:
  - name: user_id
    description: ''
    tests:
    - not_null
    - unique
  - name: some_col
    description: ''
  - name: some_other_col
    description: ''

In this case, @data-team is our default, which we sometimes replace with an individual owner, eg. @david, if the data model is very particular. This default value, we want it on every single data models that we create, and it was a little tedious to update all newly created .yml file with the property.

This PR adds an optional dbt_invoke_template.yml that will add default properties to the files updated by dbt-invoke.

What changes:

  1. load the contents of dbt_invoke_template.yml in _util.py,
  2. update the _structure_property_file_dict to apply the template's default properties.

I've also updated the README.md file so you can better understand how this works, and I've tested this by updating my local dbt-invoke package.

(feel free to edit anything on this; I'd love to have this functionality, so I'm saving the frustration of manually adding meta.owner)

david-whimsical avatar Sep 15 '23 16:09 david-whimsical

Given that this feature is totally new to dbt-invoke, we are going to need to add some tests, most likely to the tests/test_properties.py file.

A few test scenarios that immediately come to mind and that we should ensure work (both for the custom default resource properties and custom default column properties) are:

  • That everything still works as is when there is no dbt_invoke_template.yml file
  • That the output matches the expected output when there is a dbt_invoke_template.yml
  • If the value of a key in dbt_invoke_template.yml is updated, and that key already exists in the property file, then the value of that key should be updated in property file next time dbt-invoke properties.update is run

Were there any interesting scenarios or issues you noticed during your testing or any other scenarios you can think of that should be tested?

Something else I am pondering: What happens if the dbt_invoke_template.yml file contains invalid yml, is empty, or contains none of the supported resource types? In your opinion, what should happen in these cases?

robastel avatar Oct 21 '23 22:10 robastel