astronomer-cosmos icon indicating copy to clipboard operation
astronomer-cosmos copied to clipboard

Render ephemeral type DBT model & no freshness DBT source with test as EmptyOperator

Open okayhooni opened this issue 8 months ago • 6 comments

Description

  • dbt models with ephemeral materialization type that serve only as CTEs should not be rendered as DbtRunOperator tasks, as they unnecessarily occupy Airflow worker slots, even for a short period.
  • Updated the logic to render these ephemeral models as EmptyOperator tasks instead, ensuring they are processed quickly by the Airflow scheduler without being assigned to an Airflow worker.
  • Similarly, DBT source nodes without a freshness check but with downstream tests are rendered as EmptyOperator to avoid unnecessary worker slot resource consumption.

Breaking Change?

No

Checklist

  • [ ] I have made corresponding changes to the documentation (if required)
  • [ ] I have added tests that prove my fix is effective or that my feature works

okayhooni avatar Mar 18 '25 16:03 okayhooni

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
Latest commit 32b15b9cfc51cf0c55bf2a7580c9c941e3b08b01
Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67dab105e5ffc20008e00547

netlify[bot] avatar Mar 18 '25 16:03 netlify[bot]

Codecov Report

:x: Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review. :white_check_mark: Project coverage is 97.41%. Comparing base (43b5400) to head (32b15b9). :warning: Report is 143 commits behind head on main.

Files with missing lines Patch % Lines
cosmos/airflow/graph.py 88.88% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1625      +/-   ##
==========================================
- Coverage   97.43%   97.41%   -0.02%     
==========================================
  Files          80       80              
  Lines        4950     4957       +7     
==========================================
+ Hits         4823     4829       +6     
- Misses        127      128       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Mar 18 '25 19:03 codecov[bot]

This PR is stale because it has been open for 30 days with no activity.

github-actions[bot] avatar Apr 19 '25 11:04 github-actions[bot]

This PR is stale because it has been open for 30 days with no activity.

github-actions[bot] avatar May 30 '25 11:05 github-actions[bot]

hi @okayhooni, thanks a lot for this PR. Really sorry, we missed reviewing this earlier. Planning to review it now, would it be possible to resolve the conflicts on this PR, please?

pankajkoti avatar Jul 11 '25 10:07 pankajkoti

@okayhooni, were you able to add unit/integration tests for these changes?

jroachgolf84 avatar Jul 14 '25 20:07 jroachgolf84