squared
squared copied to clipboard
Environment `userdev` doesn't work for team members wanting to do dbt development
I tried to user userdev
to just run meltano run dbt-snowflake:run
and there were tons of issue that blocked me from:
- Testing existing dbt transformations.
- Adding additional dbt tranforms.
I'm opening a PR with fixes for the blockers I ran into...
The transform in cloud_ip_ranges.sql
fails. I'm not sure yet why.
I've got most things working now in #580, but many build steps are slow and some haven't been able to complete yet.
I wanted to get the full project built before adding new transforms, but ran out of time. Might be worth a 1:1 @pnadolny13 if we can find time, or else I could bring to data office hours.
@aaronsteers I commented in the PR but can you share more of the exact issues you were running into? Its totally possible that things arent working as expected because I've mostly been the only one using this repo so far so I havent tested the onboarding too much.
Thanks, @pnadolny13. I think the PR piece-by-piece resolved my issues. Maybe one last thing that would be helpful would be a section under CONTRIBUTING.md
that highlights how to run only a subset of the transforms - or conversely, how to exclude the most expensive ones.
I know there are a number of commands
registered with different --select
filters - I didn't see an obvious path for how to use the right one (or create a new one) for my use case, but admittedly this might also be a reflection of my stale dbt prowess. 😄
@aaronsteers yeah I can write up a better contributing section. It might end up being helpful to use tags for some of the larger tables so they can easily be excluded.
I know there are a number of commands registered with different --select filters - I didn't see an obvious path for how to use the right one (or create a new one) for my use case, but admittedly this might also be a reflection of my stale dbt prowess. 😄
I can add this to do docs but...when I run locally I usually use invoke
and pass select filters as usual. I only create custom meltano command if I want to schedule a job, at that point I create a command and use it in a job/schedule.