hbase icon indicating copy to clipboard operation
hbase copied to clipboard
verified publisher icon apache

HBASE-29084: Track table level metrics on HBase client side

Open sanjeet006py opened this issue 11 months ago • 5 comments

Summary of the change:

  • Introduced new class DelegateTableRunnableWithMetrics to wrap a runnable and track metrics like execution time and queue wait time. Wrapped existing runnable used by HTable#batch() API within DelegateTableRunnableWithMetrics.
  • Introduced class TableThreadMetricsHolder to hold metrics for a runnable. There is one-to-one mapping b/w instance of DelegateTableRunnableWithMetrics and TableThreadMetricsHolder.
  • Introduced class TableMetrics to collect TableThreadMetricsHolder objects. There is one TableMetrics object per HTable object.
  • Exposes list of TableThreadMetricsHolder objects by clients so, that clients can aggregate the metrics as per their requirement. Thus, giving them flexibility. But have provided one basic implementation of aggregating metrics from TableThreadMetricsHolder objects list.

Looking for inputs/suggestions around:

  • Overall design of the changes.
  • TableMetrics uses synchronized list to store TableThreadMetricsHolder objects. Understanding is that its very unlikely that multiple threads are trying to add TableThreadMetricsHolder objects to the list simultaneously. But wondering if this can cause batch calls to see increase in latency and any better alternatives?
  • Introduced a config hbase.client.table.metrics.enable to skip wrapping runnable inside DelegateTableRunnableWithMetrics. Not sure why anyone would want to skip wrapping the runnable given whether to consume metrics is totally upto end user. Maybe we can remove config?

Pending:

  • Adding test coverage.

sanjeet006py avatar Jan 15 '25 05:01 sanjeet006py

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 47s Docker mode activated.
-0 :warning: yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 :ok: mvndep 0m 16s Maven dependency ordering for branch
+1 :green_heart: mvninstall 2m 49s branch-2.6 passed
+1 :green_heart: compile 0m 36s branch-2.6 passed
+1 :green_heart: javadoc 0m 34s branch-2.6 passed
+1 :green_heart: shadedjars 5m 30s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 14s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 58s the patch passed
+1 :green_heart: compile 0m 37s the patch passed
+1 :green_heart: javac 0m 37s the patch passed
+1 :green_heart: javadoc 0m 31s the patch passed
+1 :green_heart: shadedjars 5m 44s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 9s hbase-common in the patch passed.
+1 :green_heart: unit 8m 7s hbase-client in the patch passed.
32m 43s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6603
JIRA Issue HBASE-29084
Optional Tests javac javadoc unit compile shadedjars
uname Linux 91b21ee5cf5e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 361b556a8242225f6207ef818841c872b74375f2
Default Java Temurin-1.8.0_412-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/testReport/
Max. process+thread count 350 (vs. ulimit of 30000)
modules C: hbase-common hbase-client U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 15 '25 06:01 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 43s Docker mode activated.
-0 :warning: yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 :ok: mvndep 0m 11s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 3s branch-2.6 passed
+1 :green_heart: compile 0m 39s branch-2.6 passed
+1 :green_heart: javadoc 0m 33s branch-2.6 passed
+1 :green_heart: shadedjars 5m 52s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 16s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 57s the patch passed
+1 :green_heart: compile 0m 40s the patch passed
+1 :green_heart: javac 0m 40s the patch passed
+1 :green_heart: javadoc 0m 32s the patch passed
+1 :green_heart: shadedjars 5m 48s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 7s hbase-common in the patch passed.
+1 :green_heart: unit 8m 3s hbase-client in the patch passed.
33m 4s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6603
JIRA Issue HBASE-29084
Optional Tests javac javadoc unit compile shadedjars
uname Linux 712135ef515e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 361b556a8242225f6207ef818841c872b74375f2
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/testReport/
Max. process+thread count 382 (vs. ulimit of 30000)
modules C: hbase-common hbase-client U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 15 '25 06:01 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 48s Docker mode activated.
-0 :warning: yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 16s branch-2.6 passed
+1 :green_heart: compile 0m 38s branch-2.6 passed
+1 :green_heart: javadoc 0m 35s branch-2.6 passed
+1 :green_heart: shadedjars 6m 24s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 17s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 10s the patch passed
+1 :green_heart: compile 0m 38s the patch passed
+1 :green_heart: javac 0m 38s the patch passed
+1 :green_heart: javadoc 0m 33s the patch passed
+1 :green_heart: shadedjars 6m 21s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 20s hbase-common in the patch passed.
+1 :green_heart: unit 8m 19s hbase-client in the patch passed.
35m 12s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6603
JIRA Issue HBASE-29084
Optional Tests javac javadoc unit compile shadedjars
uname Linux cc7019e5e8c7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 361b556a8242225f6207ef818841c872b74375f2
Default Java Eclipse Adoptium-11.0.23+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/testReport/
Max. process+thread count 366 (vs. ulimit of 30000)
modules C: hbase-common hbase-client U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 15 '25 06:01 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ branch-2.6 Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 47s branch-2.6 passed
+1 :green_heart: compile 1m 54s branch-2.6 passed
+1 :green_heart: checkstyle 0m 39s branch-2.6 passed
+1 :green_heart: spotbugs 1m 58s branch-2.6 passed
+1 :green_heart: spotless 0m 56s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 17s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 58s the patch passed
+1 :green_heart: compile 2m 15s the patch passed
+1 :green_heart: javac 2m 15s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 22s /results-checkstyle-hbase-common.txt hbase-common: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
-0 :warning: checkstyle 0m 29s /results-checkstyle-hbase-client.txt hbase-client: The patch generated 6 new + 13 unchanged - 0 fixed = 19 total (was 13)
+1 :green_heart: xmllint 0m 0s No new issues.
+1 :green_heart: spotbugs 2m 17s the patch passed
+1 :green_heart: hadoopcheck 18m 24s Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
-1 :x: spotless 0m 17s patch has 28 errors when running spotless:check, run spotless:apply to fix.
_ Other Tests _
+1 :green_heart: asflicense 0m 20s The patch does not generate ASF License warnings.
41m 10s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6603
JIRA Issue HBASE-29084
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless xmllint
uname Linux 5a95ce484ef6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2.6 / 361b556a8242225f6207ef818841c872b74375f2
Default Java Eclipse Adoptium-11.0.23+9
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 78 (vs. ulimit of 30000)
modules C: hbase-common hbase-client U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6603/1/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jan 15 '25 06:01 Apache-HBase

Have you looked at MetricsConnection? e.g.

  • https://github.com/apache/hbase/blob/branch-2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java
  • https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java#L97
  • https://github.com/apache/hbase/blob/branch-2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java#L829

apurtell avatar Jan 15 '25 18:01 apurtell