squared icon indicating copy to clipboard operation
squared copied to clipboard

Environment `userdev` doesn't work for team members wanting to do dbt development

Open aaronsteers opened this issue 1 year ago • 5 comments

I tried to user userdev to just run meltano run dbt-snowflake:run and there were tons of issue that blocked me from:

  1. Testing existing dbt transformations.
  2. Adding additional dbt tranforms.

I'm opening a PR with fixes for the blockers I ran into...

aaronsteers avatar Feb 27 '23 03:02 aaronsteers

The transform in cloud_ip_ranges.sql fails. I'm not sure yet why.

aaronsteers avatar Feb 27 '23 03:02 aaronsteers

I've got most things working now in #580, but many build steps are slow and some haven't been able to complete yet.

I wanted to get the full project built before adding new transforms, but ran out of time. Might be worth a 1:1 @pnadolny13 if we can find time, or else I could bring to data office hours.

aaronsteers avatar Feb 27 '23 17:02 aaronsteers

@aaronsteers I commented in the PR but can you share more of the exact issues you were running into? Its totally possible that things arent working as expected because I've mostly been the only one using this repo so far so I havent tested the onboarding too much.

pnadolny13 avatar Feb 28 '23 17:02 pnadolny13

Thanks, @pnadolny13. I think the PR piece-by-piece resolved my issues. Maybe one last thing that would be helpful would be a section under CONTRIBUTING.md that highlights how to run only a subset of the transforms - or conversely, how to exclude the most expensive ones.

I know there are a number of commands registered with different --select filters - I didn't see an obvious path for how to use the right one (or create a new one) for my use case, but admittedly this might also be a reflection of my stale dbt prowess. 😄

aaronsteers avatar Mar 01 '23 06:03 aaronsteers

@aaronsteers yeah I can write up a better contributing section. It might end up being helpful to use tags for some of the larger tables so they can easily be excluded.

I know there are a number of commands registered with different --select filters - I didn't see an obvious path for how to use the right one (or create a new one) for my use case, but admittedly this might also be a reflection of my stale dbt prowess. 😄

I can add this to do docs but...when I run locally I usually use invoke and pass select filters as usual. I only create custom meltano command if I want to schedule a job, at that point I create a command and use it in a job/schedule.

pnadolny13 avatar Mar 01 '23 17:03 pnadolny13