hive HIVE-28549: Limit the maximum number of operators merged by SharedWorkOptimizer

What changes were proposed in this pull request?

This PR would limit the maximum number of table scan operators which SWO tries to merge.

https://issues.apache.org/jira/browse/HIVE-28549

Why are the changes needed?

We observed SWO makes a negative impact when it merges too many, e.g. 50, operators. If operators are memory intensive, they might throw OOM or might slow down.

I believe we can resolve OOM with the following patch, but we still want an upper limit so that we can tune concurrency or RAM per operator reasonably. https://github.com/apache/hive/pull/5478

Does this PR introduce any user-facing change?

No.

Is the change a dependency upgrade?

No.

How was this patch tested?

I added a qtest

Oct 08 '24 09:10 okumin

Quality Gate passed

Issues
12 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Oct 08 '24 11:10 sonarqubecloud[bot]

Retriggered CI just in case

Nov 18 '24 11:11 okumin

I am checking why partition_explain_ddl always fails. In my impression, this PR is unrelated. P.S. I found it failed on the latest master branch

Nov 20 '24 07:11 okumin

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Nov 28 '24 14:11 sonarqubecloud[bot]

CI passed after merging https://github.com/apache/hive/pull/5555

Nov 30 '24 04:11 okumin

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the [email protected] list if the patch is in need of reviews.