hbase icon indicating copy to clipboard operation
hbase copied to clipboard

HBASE-28783 Concurrent execution of normalizer operations on tables in RegionNormalizerWorker

Open wangxin97-02 opened this issue 1 year ago • 2 comments

Recently, I have been managing the large tables in the HBase cluster by enabling the normalizer to set the size of the regions and keep the number of regions within a reasonable range. The current code retrieves tables from RegionNormalizerWorkQueue and performs normalization operations on the tables in series. When there are multiple large tables in a cluster that need to be managed, the efficiency will be very low. I have confirmed that each split or merge plan generated by each table during the normalizer process will be limited by RateLimiter, so I think it is reasonable to perform table normalizers concurrently. In terms of implementation, create a thread pool for executing tasks in RegionNormalizerWorker, with a default value of 1 for the number of thread pools, and provide a parameter that can be configured to other values.

wangxin97-02 avatar Aug 14 '24 08:08 wangxin97-02

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 :ok: mvndep 0m 9s Maven dependency ordering for branch
+1 :green_heart: mvninstall 2m 55s master passed
+1 :green_heart: compile 3m 28s master passed
+1 :green_heart: checkstyle 0m 47s master passed
+1 :green_heart: spotbugs 1m 56s master passed
+1 :green_heart: spotless 0m 42s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 10s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 46s the patch passed
+1 :green_heart: compile 3m 28s the patch passed
+1 :green_heart: javac 3m 28s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 33s /results-checkstyle-hbase-server.txt hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 :green_heart: spotbugs 2m 9s the patch passed
+1 :green_heart: hadoopcheck 10m 25s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
-1 :x: spotless 0m 34s patch has 40 errors when running spotless:check, run spotless:apply to fix.
_ Other Tests _
+1 :green_heart: asflicense 0m 17s The patch does not generate ASF License warnings.
37m 49s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6155
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux c7db9c297763 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 773422cc0b199cda6658f757621538e0cdb9c7a7
Default Java Eclipse Adoptium-17.0.11+9
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 84 (vs. ulimit of 30000)
modules C: hbase-common hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 14 '24 09:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 38s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 :ok: mvndep 0m 9s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 1s master passed
+1 :green_heart: compile 1m 19s master passed
+1 :green_heart: javadoc 0m 48s master passed
+1 :green_heart: shadedjars 5m 19s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 56s the patch passed
+1 :green_heart: compile 1m 40s the patch passed
+1 :green_heart: javac 1m 40s the patch passed
+1 :green_heart: javadoc 1m 9s the patch passed
+1 :green_heart: shadedjars 6m 53s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 41s hbase-common in the patch passed.
+1 :green_heart: unit 241m 25s hbase-server in the patch passed.
273m 16s
Subsystem Report/Notes
Docker ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6155
Optional Tests javac javadoc unit compile shadedjars
uname Linux f8ba1c8bc2f2 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 773422cc0b199cda6658f757621538e0cdb9c7a7
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/testReport/
Max. process+thread count 4624 (vs. ulimit of 30000)
modules C: hbase-common hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6155/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 14 '24 13:08 Apache-HBase