astronomer-cosmos
astronomer-cosmos copied to clipboard
Add default source nodes rendering
Description
Re-Opening of PR #661 I'm aiming to give a default behavior to source nodes for checking their source freshness and their tests. At the moment only for Local and VirtualEnv Operators.
One of the main limitations I found while using the custom_callback
functions on source nodes to check freshness is that nodes were being created on100% of sources but not all of them required freshness checks, this made workers waste compute time.
I'm adding a new variable into the DbtNode class called has_freshness which would be True for sources with freshness checks and False for any other resource type.
All sources with the has_freshness == False will be rendered as Empty Operators, to keep the dbt's behavior of showing sources as suggested in issue #630
Some notes: Rendered Templates don't work for source freshness nodes, I couldn't find any place where dbt was compiling the freshness sql query
Related Issue(s)
Closes: #630 Closes: #572 Closes: https://github.com/astronomer/astronomer-cosmos/issues/875
Breaking Change?
This adds a new node type (source) as a default, which changes some test behavior.
This PR also updates the dev dbt project jaffle_shop to include source nodes.
As seen in the image, source nodes with freshness checks are rendered with a blue color, while the ones rendered as EmptyOperator show a white/light green color
Checklist
- [ ] I have made corresponding changes to the documentation (if required)
- [ ] I have added tests that prove my fix is effective or that my feature works