[KYUUBI #6560] Support removing user-specified repartition before writing when using zorder

[huangxiaopingRD]

:mag: Description

Issue References 🔗

This pull request fixes #6560

Describe Your Solution 🔧

• Add RemoveRepartitionBeforeInsertInto to remove the user-specified repartition

Types of changes :bookmark:

• [ ] Bugfix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request :coffin:

Because the user added unreasonable repartition, the compression rate will not be high enough after turning on zorder, and the overall time consumption will increase significantly.

Behavior With This Pull Request :tada:

After removing the user-specified repartition, the parallelism of Rebalance is more reasonable, resulting in higher compression rate and shorter execution time.

Related Unit Tests

ZorderSuiteBase / test("Check remove user specify repartition as expected")

---

Checklist 📝

• [x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (d3e1768) to head (3bff311). Report is 109 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6561   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         677     677           
  Lines       41907   41907           
  Branches     5721    5721           
======================================
  Misses      41907   41907           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles

[pan3793]

Generally, we should always respect user explicitly requested data distribution to unsurprise users. My view may be too pessimistic, breaking user requested data distribution may cause correctness issue in corner cases, I'm -0 on introduing this feature. But I'm OK if other committers accept it, as long as we disable it by default and explain the dangerous in the docs

[huangxiaopingRD]

Generally, we should always respect user explicitly requested data distribution to unsurprise users. My view may be too pessimistic, breaking user requested data distribution may cause correctness issue in corner cases, I'm -0 on introduing this feature. But I'm OK if other committers accept it, as long as we disable it by default and explain the dangerous in the docs

Your concern is correct. This feature is only decided by the user, not by the platform. This feature is similar to Spark's RemoveAllHints

[github-actions[bot]]

Thanks for the PR! This PR is being closed due to inactivity. This isn't a judgement on the merit of the PR in any way. If this is still an issue with the latest version of Kyuubi, please reopen it and ask a committer to remove the Stale tag!

Thank you for using Kyuubi!