pinot icon indicating copy to clipboard operation
pinot copied to clipboard

[spool] Define equivalence between stages

Open gortiz opened this issue 1 year ago • 2 comments

This PR includes the code that defines when two stages are equivalent and it is the first step to implement #14196.

Instead of implementing the ability to reuse common expressions in a single and very large PR, I decided to create this first one where:

  • The new code is not being called by production code (yet)
  • Classes are well tested and documented

My hope is that this will be easier to review.

gortiz avatar Oct 24 '24 14:10 gortiz

cc @bziobrowski @vrajat

gortiz avatar Oct 24 '24 14:10 gortiz

Codecov Report

Attention: Patch coverage is 44.60784% with 113 lines in your changes missing coverage. Please review.

Project coverage is 63.75%. Comparing base (59551e4) to head (7758934). Report is 1307 commits behind head on master.

Files with missing lines Patch % Lines
.../query/planner/logical/EquivalentStagesFinder.java 40.17% 52 Missing and 18 partials :warning:
.../pinot/query/planner/plannode/PlanNodeVisitor.java 40.00% 20 Missing and 1 partial :warning:
...che/pinot/query/planner/logical/GroupedStages.java 75.00% 7 Missing and 3 partials :warning:
...anner/logical/ParentToChildrenStageCalculator.java 0.00% 8 Missing :warning:
.../apache/pinot/query/planner/plannode/PlanNode.java 0.00% 3 Missing :warning:
.../pinot/query/planner/plannode/MailboxSendNode.java 0.00% 1 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #14296      +/-   ##
============================================
+ Coverage     61.75%   63.75%   +2.00%     
- Complexity      207     1569    +1362     
============================================
  Files          2436     2665     +229     
  Lines        133233   146223   +12990     
  Branches      20636    22432    +1796     
============================================
+ Hits          82274    93221   +10947     
- Misses        44911    46087    +1176     
- Partials       6048     6915     +867     
Flag Coverage Δ
custom-integration1 100.00% <ø> (+99.99%) :arrow_up:
integration 100.00% <ø> (+99.99%) :arrow_up:
integration1 100.00% <ø> (+99.99%) :arrow_up:
integration2 0.00% <ø> (ø)
java-11 63.71% <44.60%> (+2.00%) :arrow_up:
java-21 63.64% <44.60%> (+2.01%) :arrow_up:
skip-bytebuffers-false 63.74% <44.60%> (+1.99%) :arrow_up:
skip-bytebuffers-true 63.59% <44.60%> (+35.87%) :arrow_up:
temurin 63.75% <44.60%> (+2.00%) :arrow_up:
unittests 63.74% <44.60%> (+2.00%) :arrow_up:
unittests1 55.51% <44.60%> (+8.62%) :arrow_up:
unittests2 34.09% <0.00%> (+6.36%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Oct 24 '24 15:10 codecov-commenter

I think this branch needs to be rebased / merged with master, all the builds are currently failing (at least one reason seems to be conflict with https://github.com/apache/pinot/pull/14294/).

yashmayya avatar Oct 29 '24 13:10 yashmayya