astronomer-cosmos
astronomer-cosmos copied to clipboard
Render ephemeral type DBT model & no freshness DBT source with test as EmptyOperator
Description
- dbt models with
ephemeralmaterialization type that serve only as CTEs should not be rendered asDbtRunOperatortasks, as they unnecessarily occupy Airflow worker slots, even for a short period. - Updated the logic to render these ephemeral models as EmptyOperator tasks instead, ensuring they are processed quickly by the Airflow scheduler without being assigned to an Airflow worker.
- Similarly, DBT source nodes without a freshness check but with downstream tests are rendered as EmptyOperator to avoid unnecessary worker slot resource consumption.
Breaking Change?
No
Checklist
- [ ] I have made corresponding changes to the documentation (if required)
- [ ] I have added tests that prove my fix is effective or that my feature works
Deploy Preview for sunny-pastelito-5ecb04 canceled.
| Name | Link |
|---|---|
| Latest commit | 32b15b9cfc51cf0c55bf2a7580c9c941e3b08b01 |
| Latest deploy log | https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67dab105e5ffc20008e00547 |
Codecov Report
:x: Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 97.41%. Comparing base (43b5400) to head (32b15b9).
:warning: Report is 143 commits behind head on main.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| cosmos/airflow/graph.py | 88.88% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #1625 +/- ##
==========================================
- Coverage 97.43% 97.41% -0.02%
==========================================
Files 80 80
Lines 4950 4957 +7
==========================================
+ Hits 4823 4829 +6
- Misses 127 128 +1
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
This PR is stale because it has been open for 30 days with no activity.
This PR is stale because it has been open for 30 days with no activity.
hi @okayhooni, thanks a lot for this PR. Really sorry, we missed reviewing this earlier. Planning to review it now, would it be possible to resolve the conflicts on this PR, please?
@okayhooni, were you able to add unit/integration tests for these changes?