dbt-invoke
dbt-invoke copied to clipboard
possibility to use templated properties to apply by default on file update
hey, first of all, thanks for building this package, it's been a pleasure using it!
Our team recently started to use what's in the meta
field, most notably to set the meta.owner
property:
version: 2
models:
- name: dim_user
description: >
Some description
meta:
owner: '@data-team'
columns:
- name: user_id
description: ''
tests:
- not_null
- unique
- name: some_col
description: ''
- name: some_other_col
description: ''
In this case, @data-team
is our default, which we sometimes replace with an individual owner, eg. @david
, if the data model is very particular. This default value, we want it on every single data models that we create, and it was a little tedious to update all newly created .yml
file with the property.
This PR adds an optional dbt_invoke_template.yml
that will add default properties to the files updated by dbt-invoke.
What changes:
- load the contents of
dbt_invoke_template.yml
in_util.py
, - update the
_structure_property_file_dict
to apply the template's default properties.
I've also updated the README.md
file so you can better understand how this works, and I've tested this by updating my local dbt-invoke package.
(feel free to edit anything on this; I'd love to have this functionality, so I'm saving the frustration of manually adding meta.owner
)
Given that this feature is totally new to dbt-invoke, we are going to need to add some tests, most likely to the tests/test_properties.py
file.
A few test scenarios that immediately come to mind and that we should ensure work (both for the custom default resource properties and custom default column properties) are:
- That everything still works as is when there is no
dbt_invoke_template.yml
file - That the output matches the expected output when there is a
dbt_invoke_template.yml
- If the value of a key in
dbt_invoke_template.yml
is updated, and that key already exists in the property file, then the value of that key should be updated in property file next timedbt-invoke properties.update
is run
Were there any interesting scenarios or issues you noticed during your testing or any other scenarios you can think of that should be tested?
Something else I am pondering: What happens if the dbt_invoke_template.yml
file contains invalid yml, is empty, or contains none of the supported resource types? In your opinion, what should happen in these cases?