HBASE-28783 Concurrent execution of normalizer operations on tables in RegionNormalizerWorker
Recently, I have been managing the large tables in the HBase cluster by enabling the normalizer to set the size of the regions and keep the number of regions within a reasonable range. The current code retrieves tables from RegionNormalizerWorkQueue and performs normalization operations on the tables in series. When there are multiple large tables in a cluster that need to be managed, the efficiency will be very low. I have confirmed that each split or merge plan generated by each table during the normalizer process will be limited by RateLimiter, so I think it is reasonable to perform table normalizers concurrently. In terms of implementation, create a thread pool for executing tasks in RegionNormalizerWorker, with a default value of 1 for the number of thread pools, and provide a parameter that can be configured to other values.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 2m 55s | master passed | |
| +1 :green_heart: | compile | 3m 28s | master passed | |
| +1 :green_heart: | checkstyle | 0m 47s | master passed | |
| +1 :green_heart: | spotbugs | 1m 56s | master passed | |
| +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 46s | the patch passed | |
| +1 :green_heart: | compile | 3m 28s | the patch passed | |
| +1 :green_heart: | javac | 3m 28s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 33s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 2m 9s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 10m 25s | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. | |
| -1 :x: | spotless | 0m 34s | patch has 40 errors when running spotless:check, run spotless:apply to fix. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | |
| 37m 49s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6155 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux c7db9c297763 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 773422cc0b199cda6658f757621538e0cdb9c7a7 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 3m 1s | master passed | |
| +1 :green_heart: | compile | 1m 19s | master passed | |
| +1 :green_heart: | javadoc | 0m 48s | master passed | |
| +1 :green_heart: | shadedjars | 5m 19s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 56s | the patch passed | |
| +1 :green_heart: | compile | 1m 40s | the patch passed | |
| +1 :green_heart: | javac | 1m 40s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 9s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 53s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 41s | hbase-common in the patch passed. | |
| +1 :green_heart: | unit | 241m 25s | hbase-server in the patch passed. | |
| 273m 16s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6155 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux f8ba1c8bc2f2 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 773422cc0b199cda6658f757621538e0cdb9c7a7 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/testReport/ |
| Max. process+thread count | 4624 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.