apache
HIVE-28490: Stop removing retainable DPP sources
What changes were proposed in this pull request?
This patch changes the implementation of SharedWorkOptimizer#findAscendantWorkOperators. For a given operator, original findAscendantWorkOperators() returns a set of operators that belongs to ancestor vertices of the vertex that given operator belongs to. Therefore, some operators that are irrelevant to the given operator also belong to the resultant set, which leads to wrong DPP source removal. The new implementation returns a set of ancestor operators of the given operator. Please check out the slides attached in the JIRA page (HIVE-28490) for further explanations.
Why are the changes needed?
In order to prevent the removal of retainable DPP sources that are still useful in runtime.
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
We tested this patch by running all TPC-DS queries in cluster. This patch contains a qfile test that reproduces the wrong DPP source removal.
Quality Gate passed
Issues
6 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
See analysis details on SonarCloud
![sonarqubecloud[bot] avatar](https://avatars.githubusercontent.com/in/12526?v=4 "Published by a pub.dev verified publisher"){kind=small}
thanks for the fix @ngsg! sorry for the late reply. I've added a few comments
@ramesh0201, @kasakrisz, could you please take a look at this PR as a second pair of eyes? thanks!
Quality Gate passed
Issues
6 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
See analysis details on SonarCloud