astronomer-cosmos
astronomer-cosmos copied to clipboard
Add default source nodes rendering
Description
I'm aiming to give a default behaviour to source nodes for checking their source freshness and their tests. One of the main limitations I found while using the custom_callback functions on source nodes to check freshness is that those node were being created to 100% of sources but not all of them required freshness checks, this made workers waste compute time.
I'm adding a new variable into the DbtNode class called has_freshness
which would be True for sources with freshness checks and False for the ones that not and any other resource type.
All sources with the has_freshness
== False will be rendered as Empty Operators, to keep the dbt's behavior of showing sources as suggested in issue #630.
Related Issue(s)
#630
Breaking Change?
Checklist
- [ ] I have made corresponding changes to the documentation (if required)
- [ ] I have added tests that prove my fix is effective or that my feature works
Deploy Preview for amazing-pothos-a3bca0 processing.
Name | Link |
---|---|
Latest commit | 99d4be756d39cc051074f5b9bbc21e6dd1671d80 |
Latest deploy log | https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/654c55a5e83e2700089f7f86 |
Hi @arojasb3, it looks like you've made great progress on this feature! Would you be able to rebase it so it can be reviewed and merged? Thanks!
Deploy Preview for sunny-pastelito-5ecb04 canceled.
Name | Link |
---|---|
Latest commit | c0e280ed89e7c512e9038259d9b50ab2a756b6c6 |
Latest deploy log | https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6693328cee35e50008d391be |
Hi @arojasb3, it looks like you've made great progress on this feature! Would you be able to rebase it so it can be reviewed and merged? Thanks!
I messed up the rebase, I'll just open a new PR
I messed up the rebase, I'll just open a new PR
No worries! Please go ahead and open a new PR; Iād be happy to review and test it.