hbase icon indicating copy to clipboard operation
hbase copied to clipboard

HBASE-27224 HFile tool statistic sampling produces misleading results

Open bbeaudreault opened this issue 3 years ago • 17 comments

Wraps the existing codahale Histogram in a KeyValueStats object. This object tracks the global min and max sizes for the statistic, since Histogram's is subject to sampling. Additionally adds a new -d argument which enables printing of detailed size range counts. This can be useful for further visualizing the distribution of sizes.

Below is an example form a real HFile. Prior to this patch, the max value shown was only 25k due to the sampling done by Histogram. Additionally, the new buckets (printed when -d is passed) make it easy to see that there are actually a few doze quite large rows in this file.

Stats:
   Key length:
               min = 43
               max = 137
              mean = 104.97
            stddev = 25.93
            median = 105.00
              75% <= 137.00
              95% <= 137.00
              98% <= 137.00
              99% <= 137.00
            99.9% <= 137.00
             count = 852277
                0 <= 10
              785 <= 50
           404643 <= 100
           446849 <= 500
   Val length:
               min = 1
               max = 2696196
              mean = 1633.66
            stddev = 4670.94
            median = 502.00
              75% <= 895.00
              95% <= 7567.30
              98% <= 16551.24
              99% <= 24558.18
            99.9% <= 52617.90
             count = 852544
              475 <= 1
           423450 <= 500
           238256 <= 1000
           139640 <= 5000
            19226 <= 10000
            29689 <= 50000
             1097 <= 100000
              620 <= 500000
               41 <= 750000
               19 <= 1000000
               31 <= 5000000
   Row size (bytes):
               min = 67
               max = 2696274
              mean = 1826.62
            stddev = 5465.38
            median = 619.50
              75% <= 1011.75
              95% <= 7299.55
              98% <= 16604.48
              99% <= 25200.41
            99.9% <= 82035.61
             count = 852277
                0 <= 50
               73 <= 100
           274776 <= 500
           359139 <= 1000
           167021 <= 5000
            19391 <= 10000
            30066 <= 50000
             1099 <= 100000
              621 <= 500000
               41 <= 750000
               19 <= 1000000
               31 <= 5000000
   Row size (columns):
               min = 1
               max = 2
              mean = 1.00
            stddev = 0.03
            median = 1.00
              75% <= 1.00
              95% <= 1.00
              98% <= 1.00
              99% <= 1.00
            99.9% <= 1.97
             count = 852277
           852010 <= 1
              267 <= 3

bbeaudreault avatar Jul 21 '22 00:07 bbeaudreault

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 4s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 9s master passed
+1 :green_heart: compile 2m 44s master passed
+1 :green_heart: checkstyle 0m 43s master passed
+1 :green_heart: spotless 0m 56s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 30s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 38s the patch passed
+1 :green_heart: compile 3m 13s the patch passed
+1 :green_heart: javac 3m 13s the patch passed
-0 :warning: checkstyle 0m 43s hbase-server: The patch generated 1 new + 8 unchanged - 1 fixed = 9 total (was 9)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 13m 23s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 47s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 35s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 10s The patch does not generate ASF License warnings.
38m 26s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 43562c56bf1b 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / da27a67a1e
Default Java AdoptOpenJDK-1.8.0_282-b08
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count 66 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 21 '22 00:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 6s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 26s master passed
+1 :green_heart: compile 2m 15s master passed
+1 :green_heart: checkstyle 0m 32s master passed
+1 :green_heart: spotless 0m 44s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 20s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 10s the patch passed
+1 :green_heart: compile 2m 13s the patch passed
+1 :green_heart: javac 2m 13s the patch passed
+1 :green_heart: checkstyle 0m 31s hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 11m 27s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 44s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 23s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 11s The patch does not generate ASF License warnings.
32m 17s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 76f8bb5e09c1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 31fc97edee
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 60 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 21 '22 13:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 38s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 28s master passed
+1 :green_heart: compile 0m 52s master passed
+1 :green_heart: shadedjars 4m 30s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 29s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 13s the patch passed
+1 :green_heart: compile 0m 55s the patch passed
+1 :green_heart: javac 0m 55s the patch passed
+1 :green_heart: shadedjars 4m 35s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 29s the patch passed
_ Other Tests _
+1 :green_heart: unit 214m 10s hbase-server in the patch passed.
236m 2s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux 401040289cfa 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 31fc97edee
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/testReport/
Max. process+thread count 2489 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 21 '22 16:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 44s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 50s master passed
+1 :green_heart: compile 1m 8s master passed
+1 :green_heart: shadedjars 5m 44s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 39s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 59s the patch passed
+1 :green_heart: compile 0m 53s the patch passed
+1 :green_heart: javac 0m 53s the patch passed
+1 :green_heart: shadedjars 6m 8s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 55s the patch passed
_ Other Tests _
+1 :green_heart: unit 226m 41s hbase-server in the patch passed.
252m 36s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux a61815954c50 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 31fc97edee
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/testReport/
Max. process+thread count 2753 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 21 '22 16:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 23s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 9s master passed
+1 :green_heart: compile 2m 47s master passed
+1 :green_heart: checkstyle 0m 41s master passed
+1 :green_heart: spotless 0m 52s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 38s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 44s the patch passed
+1 :green_heart: compile 2m 40s the patch passed
+1 :green_heart: javac 2m 40s the patch passed
+1 :green_heart: checkstyle 0m 32s hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 13m 49s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 48s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 44s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 11s The patch does not generate ASF License warnings.
38m 54s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 8c54007870f6 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 9ba868d2ad
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 26 '22 22:07 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 36s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 26s master passed
+1 :green_heart: compile 0m 35s master passed
+1 :green_heart: shadedjars 4m 0s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 14s the patch passed
+1 :green_heart: compile 0m 35s the patch passed
+1 :green_heart: javac 0m 35s the patch passed
+1 :green_heart: shadedjars 4m 3s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
-1 :x: unit 206m 34s hbase-server in the patch failed.
223m 10s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux bc7b6a1a5f8d 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 9ba868d2ad
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/testReport/
Max. process+thread count 2400 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 27 '22 01:07 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 5s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 58s master passed
+1 :green_heart: compile 0m 47s master passed
+1 :green_heart: shadedjars 3m 50s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 28s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 39s the patch passed
+1 :green_heart: compile 0m 48s the patch passed
+1 :green_heart: javac 0m 48s the patch passed
+1 :green_heart: shadedjars 3m 48s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s the patch passed
_ Other Tests _
-1 :x: unit 218m 9s hbase-server in the patch failed.
236m 47s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux 22268db5610a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 9ba868d2ad
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/testReport/
Max. process+thread count 2464 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 27 '22 01:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 53s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 1s master passed
+1 :green_heart: compile 2m 59s master passed
+1 :green_heart: checkstyle 0m 43s master passed
+1 :green_heart: spotless 0m 59s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 58s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 47s the patch passed
+1 :green_heart: compile 2m 39s the patch passed
+1 :green_heart: javac 2m 39s the patch passed
+1 :green_heart: checkstyle 0m 33s hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 13m 58s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 50s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 38s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 7s The patch does not generate ASF License warnings.
41m 1s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux f00692c15f69 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 80b366817e
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 27 '22 04:07 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 37s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 27s master passed
+1 :green_heart: compile 0m 36s master passed
+1 :green_heart: shadedjars 4m 2s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 12s the patch passed
+1 :green_heart: compile 0m 35s the patch passed
+1 :green_heart: javac 0m 35s the patch passed
+1 :green_heart: shadedjars 4m 2s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
-1 :x: unit 202m 27s hbase-server in the patch failed.
219m 7s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux 9745001003a3 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 80b366817e
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/testReport/
Max. process+thread count 2617 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 27 '22 07:07 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 56s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 4m 24s master passed
+1 :green_heart: compile 1m 9s master passed
+1 :green_heart: shadedjars 5m 3s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 37s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 21s the patch passed
+1 :green_heart: compile 1m 7s the patch passed
+1 :green_heart: javac 1m 7s the patch passed
+1 :green_heart: shadedjars 5m 15s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 30s the patch passed
_ Other Tests _
-1 :x: unit 213m 7s hbase-server in the patch failed.
238m 16s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux b9d665c321aa 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 80b366817e
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/testReport/
Max. process+thread count 2808 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 27 '22 08:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 10s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 16s master passed
+1 :green_heart: compile 2m 12s master passed
+1 :green_heart: checkstyle 0m 30s master passed
+1 :green_heart: spotless 0m 43s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 17s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 10s the patch passed
+1 :green_heart: compile 2m 13s the patch passed
+1 :green_heart: javac 2m 13s the patch passed
+1 :green_heart: checkstyle 0m 30s hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 11m 22s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 42s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 24s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 10s The patch does not generate ASF License warnings.
31m 28s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 7bb663f5d951 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 811f0e7d2a
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 29 '22 13:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 15s Docker mode activated.
-0 :warning: yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 34s master passed
+1 :green_heart: compile 0m 46s master passed
+1 :green_heart: shadedjars 3m 43s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 25s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 34s the patch passed
+1 :green_heart: compile 0m 47s the patch passed
+1 :green_heart: javac 0m 47s the patch passed
+1 :green_heart: shadedjars 3m 42s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s the patch passed
_ Other Tests _
+1 :green_heart: unit 202m 40s hbase-server in the patch passed.
220m 24s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux 7423efb4d9c0 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 811f0e7d2a
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/testReport/
Max. process+thread count 2644 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 29 '22 16:07 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 40s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 12s master passed
+1 :green_heart: compile 0m 35s master passed
+1 :green_heart: shadedjars 4m 7s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 23s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 17s the patch passed
+1 :green_heart: compile 0m 36s the patch passed
+1 :green_heart: javac 0m 36s the patch passed
+1 :green_heart: shadedjars 4m 5s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
+1 :green_heart: unit 204m 24s hbase-server in the patch passed.
221m 4s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux a4fabe2db5af 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 811f0e7d2a
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/testReport/
Max. process+thread count 2551 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Jul 29 '22 16:07 Apache-HBase

@cbaenziger thanks for the input. I made changes per your request. Here's the updated format:

   Key length:
               min = 29
               max = 29
              mean = 29.00
            stddev = 0.00
            median = 29.00
              75% <= 29.00
              95% <= 29.00
              98% <= 29.00
              99% <= 29.00
            99.9% <= 29.00
             count = 1000
           (range <= count):
               10 <= 0
               50 <= 1000
   Val length:
               min = 3
               max = 3
              mean = 3.00
            stddev = 0.00
            median = 3.00
              75% <= 3.00
              95% <= 3.00
              98% <= 3.00
              99% <= 3.00
            99.9% <= 3.00
             count = 1000
           (range <= count):
                1 <= 0
                3 <= 1000
   Row size (bytes):
               min = 40
               max = 40
              mean = 40.00
            stddev = 0.00
            median = 40.00
              75% <= 40.00
              95% <= 40.00
              98% <= 40.00
              99% <= 40.00
            99.9% <= 40.00
             count = 1000
           (range <= count):
               10 <= 0
               50 <= 1000
   Row size (columns):
               min = 1
               max = 1
              mean = 1.00
            stddev = 0.00
            median = 1.00
              75% <= 1.00
              95% <= 1.00
              98% <= 1.00
              99% <= 1.00
            99.9% <= 1.00
             count = 1000
           (range <= count):
                1 <= 1000


Key of biggest row: row_00000000

Worth noting that the new range <= count sections only show up if you enable the new -d arg.

To your last question:

Lastly, this may be a question/solution chasing a problem, but will this show a bi-modal set of ranges clearly (e.g. if I have keys of 50 bytes and keys of 5,000 bytes only) or will the elided bounding ranges be needed to point that out? Or is that what line 845 is doing already? Yes, line 845 attempts to provide some context while not printing all of the 0 ranges between 2 values. Probably the most clear would be to print all ranges, but that'd also be way unnecessarily verbose in most cases. So I tried to only print the context necessary, i think it should be pretty intuitive once someone sees the output for a few files.

bbeaudreault avatar Aug 08 '22 13:08 bbeaudreault

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 11s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 20s master passed
+1 :green_heart: compile 2m 18s master passed
+1 :green_heart: checkstyle 0m 31s master passed
+1 :green_heart: spotless 0m 43s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 17s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 15s the patch passed
+1 :green_heart: compile 2m 10s the patch passed
+1 :green_heart: javac 2m 10s the patch passed
+1 :green_heart: checkstyle 0m 30s hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 11m 25s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 41s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 22s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 10s The patch does not generate ASF License warnings.
32m 3s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 2f5574db23eb 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 54f2106f33
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 08 '22 14:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 48s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 31s master passed
+1 :green_heart: compile 0m 34s master passed
+1 :green_heart: shadedjars 4m 1s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 7s the patch passed
+1 :green_heart: compile 0m 34s the patch passed
+1 :green_heart: javac 0m 34s the patch passed
+1 :green_heart: shadedjars 4m 3s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
+1 :green_heart: unit 201m 16s hbase-server in the patch passed.
218m 33s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux df5bb9638448 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 54f2106f33
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/testReport/
Max. process+thread count 2976 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 08 '22 17:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 7s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 44s master passed
+1 :green_heart: compile 0m 46s master passed
+1 :green_heart: shadedjars 3m 45s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 38s the patch passed
+1 :green_heart: compile 0m 46s the patch passed
+1 :green_heart: javac 0m 46s the patch passed
+1 :green_heart: shadedjars 3m 44s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s the patch passed
_ Other Tests _
+1 :green_heart: unit 203m 17s hbase-server in the patch passed.
221m 32s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4638
Optional Tests javac javadoc unit shadedjars compile
uname Linux d906e0d032c9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 54f2106f33
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/testReport/
Max. process+thread count 2840 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 08 '22 17:08 Apache-HBase