astronomer-cosmos icon indicating copy to clipboard operation
astronomer-cosmos copied to clipboard

Add default source nodes rendering

Open arojasb3 opened this issue 1 year ago ā€¢ 1 comments

Description

I'm aiming to give a default behaviour to source nodes for checking their source freshness and their tests. One of the main limitations I found while using the custom_callback functions on source nodes to check freshness is that those node were being created to 100% of sources but not all of them required freshness checks, this made workers waste compute time.

I'm adding a new variable into the DbtNode class called has_freshness which would be True for sources with freshness checks and False for the ones that not and any other resource type.

All sources with the has_freshness == False will be rendered as Empty Operators, to keep the dbt's behavior of showing sources as suggested in issue #630.

Related Issue(s)

#630

Breaking Change?

Checklist

  • [ ] I have made corresponding changes to the documentation (if required)
  • [ ] I have added tests that prove my fix is effective or that my feature works

arojasb3 avatar Nov 09 '23 03:11 arojasb3

Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
Latest commit 99d4be756d39cc051074f5b9bbc21e6dd1671d80
Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/654c55a5e83e2700089f7f86

netlify[bot] avatar Nov 09 '23 03:11 netlify[bot]

Hi @arojasb3, it looks like you've made great progress on this feature! Would you be able to rebase it so it can be reviewed and merged? Thanks!

pankajastro avatar Jul 12 '24 12:07 pankajastro

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
Latest commit c0e280ed89e7c512e9038259d9b50ab2a756b6c6
Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6693328cee35e50008d391be

netlify[bot] avatar Jul 14 '24 02:07 netlify[bot]

Hi @arojasb3, it looks like you've made great progress on this feature! Would you be able to rebase it so it can be reviewed and merged? Thanks!

I messed up the rebase, I'll just open a new PR

arojasb3 avatar Jul 14 '24 02:07 arojasb3

I messed up the rebase, I'll just open a new PR

No worries! Please go ahead and open a new PR; Iā€™d be happy to review and test it.

pankajastro avatar Jul 14 '24 19:07 pankajastro