astronomer-cosmos icon indicating copy to clipboard operation
astronomer-cosmos copied to clipboard

Add default source nodes rendering

Open arojasb3 opened this issue 7 months ago • 14 comments

Description

Re-Opening of PR #661 I'm aiming to give a default behavior to source nodes for checking their source freshness and their tests. At the moment only for Local and VirtualEnv Operators.

One of the main limitations I found while using the custom_callback functions on source nodes to check freshness is that nodes were being created on100% of sources but not all of them required freshness checks, this made workers waste compute time.

I'm adding a new variable into the DbtNode class called has_freshness which would be True for sources with freshness checks and False for any other resource type.

All sources with the has_freshness == False will be rendered as Empty Operators, to keep the dbt's behavior of showing sources as suggested in issue #630

Some notes: Rendered Templates don't work for source freshness nodes, I couldn't find any place where dbt was compiling the freshness sql query

Related Issue(s)

Closes: #630 Closes: #572 Closes: https://github.com/astronomer/astronomer-cosmos/issues/875

Breaking Change?

This adds a new node type (source) as a default, which changes some test behavior. This PR also updates the dev dbt project jaffle_shop to include source nodes. image

As seen in the image, source nodes with freshness checks are rendered with a blue color, while the ones rendered as EmptyOperator show a white/light green color

Checklist

  • [ ] I have made corresponding changes to the documentation (if required)
  • [ ] I have added tests that prove my fix is effective or that my feature works

arojasb3 avatar Jul 19 '24 02:07 arojasb3