hbase icon indicating copy to clipboard operation
hbase copied to clipboard

HBASE-27523 Add BulkLoad bandwidth throttling

Open SiCheng-Zheng opened this issue 2 years ago • 11 comments

SiCheng-Zheng avatar Jan 16 '23 16:01 SiCheng-Zheng

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 7s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 38s master passed
+1 :green_heart: compile 2m 26s master passed
+1 :green_heart: checkstyle 0m 37s master passed
+1 :green_heart: spotless 0m 43s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 29s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 27s the patch passed
+1 :green_heart: compile 2m 26s the patch passed
+1 :green_heart: javac 2m 26s the patch passed
-0 :warning: checkstyle 0m 35s hbase-server: The patch generated 5 new + 36 unchanged - 0 fixed = 41 total (was 36)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 8m 47s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 :x: spotless 0m 36s patch has 68 errors when running spotless:check, run spotless:apply to fix.
+1 :green_heart: spotbugs 1m 36s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 12s The patch does not generate ASF License warnings.
31m 21s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux f99e104b4986 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 7ed2cb99f9
Default Java Eclipse Adoptium-11.0.17+8
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 79 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 16 '23 16:01 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 35s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 38s master passed
+1 :green_heart: compile 0m 41s master passed
+1 :green_heart: shadedjars 3m 58s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 23s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 26s the patch passed
+1 :green_heart: compile 0m 42s the patch passed
+1 :green_heart: javac 0m 42s the patch passed
+1 :green_heart: shadedjars 3m 57s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
-1 :x: unit 200m 26s hbase-server in the patch failed.
219m 58s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8f5ab45ca3c3 5.4.0-1093-aws #102~18.04.2-Ubuntu SMP Wed Dec 7 00:31:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 7ed2cb99f9
Default Java Eclipse Adoptium-11.0.17+8
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/testReport/
Max. process+thread count 2530 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 16 '23 20:01 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 42s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 30s master passed
+1 :green_heart: compile 0m 36s master passed
+1 :green_heart: shadedjars 4m 4s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 23s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 6s the patch passed
+1 :green_heart: compile 0m 36s the patch passed
+1 :green_heart: javac 0m 36s the patch passed
+1 :green_heart: shadedjars 4m 4s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
+1 :green_heart: unit 208m 59s hbase-server in the patch passed.
228m 56s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests javac javadoc unit shadedjars compile
uname Linux bccb61b9a0f3 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 7ed2cb99f9
Default Java Temurin-1.8.0_352-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/testReport/
Max. process+thread count 2634 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 16 '23 20:01 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 15m 36s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 22s master passed
+1 :green_heart: compile 2m 22s master passed
+1 :green_heart: checkstyle 0m 36s master passed
+1 :green_heart: spotless 0m 44s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 28s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 19s the patch passed
+1 :green_heart: compile 2m 23s the patch passed
+1 :green_heart: javac 2m 23s the patch passed
-0 :warning: checkstyle 0m 35s hbase-server: The patch generated 4 new + 36 unchanged - 0 fixed = 40 total (was 36)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 12m 9s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 :x: spotless 0m 36s patch has 69 errors when running spotless:check, run spotless:apply to fix.
-1 :x: spotbugs 1m 37s hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
_ Other Tests _
+1 :green_heart: asflicense 0m 12s The patch does not generate ASF License warnings.
52m 3s
Reason Tests
FindBugs module:hbase-server
Inconsistent synchronization of org.apache.hadoop.hbase.regionserver.throttle.BulkLoadThrottler.cycleStartTick; locked 75% of time Unsynchronized access at BulkLoadThrottler.java:75% of time Unsynchronized access at BulkLoadThrottler.java:[line 138]
org.apache.hadoop.hbase.regionserver.throttle.BulkLoadThrottler.limitNextBytes(int) calls Thread.sleep() with a lock held At BulkLoadThrottler.java:lock held At BulkLoadThrottler.java:[line 104]
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 6ac6969bac47 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 8e72cc6ee0
Default Java Eclipse Adoptium-11.0.17+8
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-general-check/output/patch-spotless.txt
spotbugs https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html
Max. process+thread count 86 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 01 '23 18:02 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 8m 22s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 32s master passed
+1 :green_heart: compile 0m 41s master passed
+1 :green_heart: shadedjars 4m 36s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 23s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 17s the patch passed
+1 :green_heart: compile 0m 42s the patch passed
+1 :green_heart: javac 0m 42s the patch passed
+1 :green_heart: shadedjars 4m 34s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
+1 :green_heart: unit 196m 16s hbase-server in the patch passed.
227m 2s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests javac javadoc unit shadedjars compile
uname Linux cd72950505dd 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 8e72cc6ee0
Default Java Eclipse Adoptium-11.0.17+8
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/testReport/
Max. process+thread count 2681 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 01 '23 21:02 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 41s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 17s master passed
+1 :green_heart: compile 0m 34s master passed
+1 :green_heart: shadedjars 4m 35s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 48s the patch passed
+1 :green_heart: compile 0m 34s the patch passed
+1 :green_heart: javac 0m 34s the patch passed
+1 :green_heart: shadedjars 4m 35s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
-1 :x: unit 221m 25s hbase-server in the patch failed.
244m 8s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests javac javadoc unit shadedjars compile
uname Linux 601ab3829d0d 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 8e72cc6ee0
Default Java Temurin-1.8.0_352-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/testReport/
Max. process+thread count 2650 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 01 '23 22:02 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 0s Docker mode activated.
-1 :x: docker 0m 1s Docker failed to build yetus/hbase:e7d1a6a052.
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hbase/pull/4975
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/3/console
versions git=2.17.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 02 '23 16:02 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 0s Docker mode activated.
-1 :x: docker 0m 0s Docker failed to build yetus/hbase:e7d1a6a052.
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hbase/pull/4975
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/3/console
versions git=2.17.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 02 '23 16:02 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 58s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 40s master passed
+1 :green_heart: compile 2m 30s master passed
+1 :green_heart: checkstyle 0m 35s master passed
+1 :green_heart: spotless 0m 43s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 32s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 33s the patch passed
+1 :green_heart: compile 2m 36s the patch passed
+1 :green_heart: javac 2m 36s the patch passed
+1 :green_heart: checkstyle 0m 35s the patch passed
+1 :green_heart: whitespace 0m 1s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 12m 40s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 :green_heart: spotless 0m 42s patch has no errors when running spotless:check.
-1 :x: spotbugs 1m 45s hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
_ Other Tests _
+1 :green_heart: asflicense 0m 8s The patch does not generate ASF License warnings.
39m 39s
Reason Tests
FindBugs module:hbase-server
Inconsistent synchronization of org.apache.hadoop.hbase.regionserver.throttle.BulkLoadThrottler.cycleStartTick; locked 75% of time Unsynchronized access at BulkLoadThrottler.java:75% of time Unsynchronized access at BulkLoadThrottler.java:[line 137]
org.apache.hadoop.hbase.regionserver.throttle.BulkLoadThrottler.limitNextBytes(int) calls Thread.sleep() with a lock held At BulkLoadThrottler.java:lock held At BulkLoadThrottler.java:[line 104]
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4975
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 7eea93d3190e 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 55b4bbc057
Default Java Eclipse Adoptium-11.0.17+8
spotbugs https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/3/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html
Max. process+thread count 86 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4975/3/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 02 '23 16:02 Apache-HBase

Can we use existing ThroughputController interface for this?

bbeaudreault avatar Feb 13 '23 14:02 bbeaudreault

ThroughputController interface

Let me study the ThroughputController interface

SiCheng-Zheng avatar Feb 15 '23 15:02 SiCheng-Zheng