dbt-core
dbt-core copied to clipboard
[CT-1034] [Bug] Unexpected behavior when chaining methods on dbt-ref'ed/sourced dataframes
Is this a new bug in dbt-core?
- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
Community Slack report here:
https://getdbt.slack.com/archives/C03QUA7DWCW/p1660223963454249
I reproduced easily -- the first run had df = df.limit(100)
line uncommented and no chained call on the prior line, the second run is what's shown in the screenshot:
Expected Behavior
Would not expect method chaining to cause errors like this on dbt.ref(...)
or dbt.source(...)
. We should clearly document this behavior and give a better error method if there's some underlying reason why we can't easily fix this, otherwise we should fix it.
Steps To Reproduce
See screenshot above. Basically add a .limit(N)
to your reference call, e.g. df = dbt.ref(model_name).limit(100)
.
Relevant log output
@lostmygithubaccount ➜ /workspaces/codyspace/nyc_taxi_with_python (main ✗) $ dbt run -s int_yellow
17:10:04 Running with dbt=1.3.0-b1
17:10:04 Found 5 models, 0 tests, 0 snapshots, 1 analysis, 571 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
17:10:04
17:10:05 Concurrency: 8 threads (target='dev')
17:10:05
17:10:05 1 of 1 START python table model dbt_cody.int_yellow ............................ [RUN]
17:10:10 1 of 1 ERROR creating python table model dbt_cody.int_yellow ................... [ERROR in 4.58s]
17:10:10
17:10:10 Finished running 1 table model in 0 hours 0 minutes and 5.57 seconds (5.57s).
17:10:10
17:10:10 Completed with 1 error and 0 warnings:
17:10:10
17:10:10 Database Error in model int_yellow (models/intermediate/int_yellow.py)
17:10:10 100357 (P0000): Python Interpreter Error:
17:10:10 Traceback (most recent call last):
17:10:10 File "_udf_code.py", line 83, in main
17:10:10 File "_udf_code.py", line 7, in model
17:10:10 File "_udf_code.py", line 53, in <lambda>
17:10:10 File "_udf_code.py", line 21, in ref
17:10:10 KeyError: 'stg_yellow'
17:10:10 in function INT_YELLOW__DBT_SP with handler main
17:10:10 compiled Code at target/run/nyc_taxi_with_python/models/intermediate/int_yellow.py
17:10:10
17:10:10 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
Environment
- OS: linux
- Python: 3.10
- dbt: 1.3 beta, installed via main branches
Which database adapter are you using with dbt?
snowflake
Additional Context
cc: @ChenyuLInx
No response