astronomer-cosmos icon indicating copy to clipboard operation
astronomer-cosmos copied to clipboard

Select models which have tests with additional dependencies

Open adammarples opened this issue 1 year ago • 4 comments

Description

Tests such as the relationship test will fail to be selected if we are selecting models with tags and using the manifest loading method.

Related Issue(s)

closes #719

Breaking Change?

Almost all tests will have only one 'depends_on', but when they have two, ie. relationship, the true parent is the last one.

load_from_dbt_manifest now considers the primary selectable parent model of a test to be the last item in its 'depends_on' list, rather than the first.

This would break any tests being loaded by tag selection using load_from_dbt_manifest which somehow depended on a model being the first in that list rather than the last.

Checklist

  • [ ] I have made corresponding changes to the documentation (if required)
  • [x] I have added tests that prove my fix is effective or that my feature works

adammarples avatar Dec 06 '23 15:12 adammarples

Deploy Preview for amazing-pothos-a3bca0 ready!

Name Link
Latest commit 213620fccdf33fe05375f65cde17bf350eb1f20e
Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/657af27701122100082a33fc
Deploy Preview https://deploy-preview-751--amazing-pothos-a3bca0.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] avatar Dec 06 '23 15:12 netlify[bot]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (82e8db9) 93.27% compared to head (6c4bfe4) 93.28%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #751      +/-   ##
==========================================
+ Coverage   93.27%   93.28%   +0.01%     
==========================================
  Files          55       55              
  Lines        2499     2503       +4     
==========================================
+ Hits         2331     2335       +4     
  Misses        168      168              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 07 '23 11:12 codecov[bot]

@adammarples as you're working on this, I'd like to bring to your attention a case you're missing, and which we're running into recently:

  File "/home/airflow/.local/lib/python3.11/site-packages/cosmos/dbt/selector.py", line 298, in _should_include_node
    node.tags = getattr(self.nodes.get(node.depends_on[0]), "tags", [])
                                       ~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Although not a dbt best practice, it's entirely possible to have a test with 0 dependencies, or model dependencies (test on sources)

ms32035 avatar Dec 20 '23 00:12 ms32035

Hi @adammarples , this PR - and the discussion - was beneficial. Thank you. And thanks, @jbandoro, for giving an alternative solution to the problem.

Cosmos can improve its implementation towards tests with multiple parents - we should make it consistent with dbt. @adammarples, you make a great start - would you be interested in further contributing to this - or would you rather close this PR and let someone else take over the work?

@ms32035 I didn't realise that was possible! Although it relates to the original problem, I logged a separate ticket: #782. Would you be interested in contributing?

tatiana avatar Jan 05 '24 12:01 tatiana

Closing this PR in favour of the ticket: https://github.com/astronomer/astronomer-cosmos/issues/978

tatiana avatar May 17 '24 10:05 tatiana