HBASE-29761: The HBase UI's Debug Dump is not redacting sensitive information
https://issues.apache.org/jira/browse/HBASE-29761
This pull request fixes an issue where the HBase UI's Debug Dump feature was still showing sensitive config information (such as passwords). Consider an hbase-site.xml file that contains the following:
<property>
<name>hbase.zookeeper.property.ssl.trustStore.password</name>
<value>kevin-zk-pw</value>
</property>
<property>
<name>ssl.client.truststore.password</name>
<value>kevin-ssl-truststore-pw</value>
</property>
<property>
<name>hbase.rpc.tls.truststore.password</name>
<value>kevin-tls-truststore-pw</value>
</property>
<property>
<name>ssl.server.keystore.password</name>
<value>kevin-ssl-keystore-pw</value>
</property>
<property>
<name>hadoop.security.sensitive-config-keys</name>
<value>
secret$
password$
hbase\.zookeeper\.property\.ssl\.trustStore\.password$
ssl.keystore.pass$
fs.s3a.server-side-encryption.key
fs.s3a.*.server-side-encryption.key
fs.s3a.encryption.algorithm
fs.s3a.encryption.key
fs.s3a.secret.key
fs.s3a.*.secret.key
fs.s3a.session.key
fs.s3a.*.session.key
fs.s3a.session.token
fs.s3a.*.session.token
fs.azure.account.key.*
fs.azure.oauth2.*
fs.adl.oauth2.*
fs.gs.encryption.*
fs.gs.proxy.*
fs.gs.auth.*
credential$
oauth.*secret
oauth.*password
oauth.*token
hadoop.security.sensitive-config-keys
</value>
</property>
Here, hadoop.security.sensitive-config-keys specifies various regexes for what config property names should have their values redacted. However, before this change, properties such as the ones listed above (hbase.zookeeper.property.ssl.trustStore.password, etc.) would still have their sensitive contents present in plain text in the HBase UI's Debug Dump.
With this change, these sensitive values are now redacted and replaced with ******. The issue was occurring because the wrong Configuration.writeXml() method was being called in MasterDumpServlet and RSDumpServlet. Before, the method being used resulted in a call chain of Configuration.writeXml() methods that eventually led to ConfigRedactor being null. This change directly calls the writeXml() method that was at the end of the call chain, which allows the ConfigRedactor to be established.
In addition, the unit tests created in this PR reused methods found in TestMasterStatusPage.java and TestRSStatusPage.java. These common methods have been moved to a new TestServerHttpUtils.java to eliminate repeated code.
Hi @PDavid, can you please review this PR? Also, I am still having trouble getting the HBase UI to work locally, so I have not been able to manually test this change when I branch off of the latest version of master.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 26s | master passed | |
| +1 :green_heart: | compile | 3m 21s | master passed | |
| +1 :green_heart: | checkstyle | 0m 59s | master passed | |
| +1 :green_heart: | spotbugs | 1m 36s | master passed | |
| +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. | |
| -0 :warning: | patch | 0m 59s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 53s | the patch passed | |
| +1 :green_heart: | compile | 3m 26s | the patch passed | |
| +1 :green_heart: | javac | 3m 26s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 58s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 39s | the patch passed | |
| -1 :x: | hadoopcheck | 1m 45s | The patch causes 28 errors with Hadoop v3.3.6. | |
| +1 :green_heart: | spotless | 0m 44s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | |
| 29m 42s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 0944472083aa 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b67f4bcb465d8368eb096af24a2d183568fff63a |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/1/artifact/yetus-general-check/output/patch-javac-3.3.6.txt |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 54s | master passed | |
| +1 :green_heart: | compile | 1m 4s | master passed | |
| +1 :green_heart: | javadoc | 0m 32s | master passed | |
| +1 :green_heart: | shadedjars | 6m 23s | branch has no errors when building our shaded downstream artifacts. | |
| -0 :warning: | patch | 6m 32s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 26s | the patch passed | |
| +1 :green_heart: | compile | 1m 6s | the patch passed | |
| +1 :green_heart: | javac | 1m 6s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 32s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 23s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 219m 56s | /patch-unit-hbase-server.txt | hbase-server in the patch failed. |
| 248m 39s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux b1bcc55c64e2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b67f4bcb465d8368eb096af24a2d183568fff63a |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/1/testReport/ |
| Max. process+thread count | 3840 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 50s | master passed | |
| +1 :green_heart: | compile | 3m 24s | master passed | |
| +1 :green_heart: | checkstyle | 0m 58s | master passed | |
| +1 :green_heart: | spotbugs | 1m 33s | master passed | |
| +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 54s | the patch passed | |
| +1 :green_heart: | compile | 3m 20s | the patch passed | |
| +1 :green_heart: | javac | 3m 20s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 57s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 41s | the patch passed | |
| -1 :x: | hadoopcheck | 1m 40s | The patch causes 28 errors with Hadoop v3.3.6. | |
| +1 :green_heart: | spotless | 0m 45s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | |
| 28m 45s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 32b14a5d9376 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3a45a845750e009c9e42c1b9b8fc5b907db37e77 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/2/artifact/yetus-general-check/output/patch-javac-3.3.6.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/2/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 52s | master passed | |
| +1 :green_heart: | compile | 1m 6s | master passed | |
| +1 :green_heart: | javadoc | 0m 33s | master passed | |
| +1 :green_heart: | shadedjars | 6m 27s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed | |
| +1 :green_heart: | compile | 1m 4s | the patch passed | |
| +1 :green_heart: | javac | 1m 4s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 21s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 220m 25s | hbase-server in the patch passed. | |
| 248m 41s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux fde3db2b51c0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3a45a845750e009c9e42c1b9b8fc5b907db37e77 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/2/testReport/ |
| Max. process+thread count | 4575 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
Hi @PDavid, can you please review this PR? Also, I am still having trouble getting the HBase UI to work locally, so I have not been able to manually test this change when I branch off of the latest version of master.
Hi @kgeisz, many thanks for fixing this. I pulled your branch, built it locally and tested the debug dump of both Master and RegionServer Status servlet and the fix is working. :+1:
However the hadoopcheck failed with Hadoop v3.3.6:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-server: Compilation failure: Compilation failure:
[ERROR] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-7545/yetus-general-check/src/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/http/RSDumpServlet.java:[94,11] no suitable method found for writeXml(<nulltype>,java.io.OutputStreamWriter,org.apache.hadoop.conf.Configuration)
[ERROR] method org.apache.hadoop.conf.Configuration.writeXml(java.io.OutputStream) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method org.apache.hadoop.conf.Configuration.writeXml(java.io.Writer) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method org.apache.hadoop.conf.Configuration.writeXml(java.lang.String,java.io.Writer) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-7545/yetus-general-check/src/hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/MasterDumpServlet.java:[94,11] no suitable method found for writeXml(<nulltype>,java.io.OutputStreamWriter,org.apache.hadoop.conf.Configuration)
[ERROR] method org.apache.hadoop.conf.Configuration.writeXml(java.io.OutputStream) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method org.apache.hadoop.conf.Configuration.writeXml(java.io.Writer) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method org.apache.hadoop.conf.Configuration.writeXml(java.lang.String,java.io.Writer) is not applicable
[ERROR] (actual and formal argument lists differ in length)
Can you please check it?
Hi @PDavid, it turned out there is a compatibility issue with Hadoop versions older than Hadoop 3.4. YARN-11308 introduced the new Configuration.writeXml() method that I was using in my original change. Since this was causing a build error, and also would not be compatible with branch-2, I decided to manually perform the redaction myself.
I introduced a new method to StateDumpServlet.java that performs the redaction. I also decided to have the redacted text be <redacted> rather than ****** (I can change it back if needed). I also added a private constructor to TestServerHttpUtils.java, which will hopefully stop the spotless error.
cc. @taklwu
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 52s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 5m 25s | master passed | |
| +1 :green_heart: | compile | 3m 38s | master passed | |
| +1 :green_heart: | checkstyle | 1m 6s | master passed | |
| +1 :green_heart: | spotbugs | 1m 50s | master passed | |
| +1 :green_heart: | spotless | 0m 51s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 10s | the patch passed | |
| +1 :green_heart: | compile | 3m 28s | the patch passed | |
| +1 :green_heart: | javac | 3m 28s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 1m 0s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 51s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 11m 43s | Patch does not cause any errors with Hadoop 3.3.6 3.4.1. | |
| +1 :green_heart: | spotless | 0m 46s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | |
| 43m 45s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 2fe49208d070 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dcb218558a9f0d905842003f9856b250d4e45597 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/3/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 50s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 5m 9s | master passed | |
| +1 :green_heart: | compile | 1m 28s | master passed | |
| +1 :green_heart: | javadoc | 0m 42s | master passed | |
| +1 :green_heart: | shadedjars | 9m 5s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 4m 24s | the patch passed | |
| +1 :green_heart: | compile | 1m 6s | the patch passed | |
| +1 :green_heart: | javac | 1m 6s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 27s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 221m 20s | hbase-server in the patch passed. | |
| 255m 50s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 4b14a63054a6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dcb218558a9f0d905842003f9856b250d4e45597 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/3/testReport/ |
| Max. process+thread count | 5191 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/3/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
Hi @PDavid, it turned out there is a compatibility issue with Hadoop versions older than Hadoop 3.4. YARN-11308 introduced the new Configuration.writeXml() method that I was using in my original change. Since this was causing a build error, and also would not be compatible with branch-2, I decided to manually perform the redaction myself.
I introduced a new method to
StateDumpServlet.javathat performs the redaction. I also decided to have the redacted text be<redacted>rather than******(I can change it back if needed). I also added a private constructor toTestServerHttpUtils.java, which will hopefully stop the spotless error.
Many thanks for looking into this @kgeisz and I think it is a good idea to do the redaction ourselves. :+1:
I'd not introduce another XML element instead of the text content we have now because if any application parses the output of the dump servlet would need to be adjusted after this change - this would break compatibility. So I'd keep the "*****" instead. What do you all think?
I'd not introduce another XML element instead of the text content we have now because if any application parses the output of the dump servlet would need to be adjusted after this change - this would break compatibility. So I'd keep the "*****" instead. What do you all think?
@PDavid That sounds good to me. I will make it so the Debug Dump has ****** instead of <redacted>.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 34s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 57s | master passed | |
| +1 :green_heart: | compile | 2m 30s | master passed | |
| +1 :green_heart: | checkstyle | 0m 45s | master passed | |
| +1 :green_heart: | spotbugs | 1m 15s | master passed | |
| +1 :green_heart: | spotless | 0m 39s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 12s | the patch passed | |
| +1 :green_heart: | compile | 2m 33s | the patch passed | |
| +1 :green_heart: | javac | 2m 33s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 1m 0s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 34s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 9m 17s | Patch does not cause any errors with Hadoop 3.3.6 3.4.1. | |
| +1 :green_heart: | spotless | 0m 36s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | |
| 33m 6s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux a9370f963883 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2aa66e3d407503ba954b9c2bc7ecd8f3b909ac24 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/4/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 30s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 3s | master passed | |
| +1 :green_heart: | compile | 0m 45s | master passed | |
| +1 :green_heart: | javadoc | 0m 22s | master passed | |
| +1 :green_heart: | shadedjars | 4m 31s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 17s | the patch passed | |
| +1 :green_heart: | compile | 0m 45s | the patch passed | |
| +1 :green_heart: | javac | 0m 45s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed | |
| +1 :green_heart: | shadedjars | 4m 28s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 193m 42s | hbase-server in the patch passed. | |
| 215m 42s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 42d62ad62bfb 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2aa66e3d407503ba954b9c2bc7ecd8f3b909ac24 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/4/testReport/ |
| Max. process+thread count | 6353 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/4/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 28s | master passed | |
| +1 :green_heart: | compile | 3m 20s | master passed | |
| +1 :green_heart: | checkstyle | 1m 0s | master passed | |
| +1 :green_heart: | spotbugs | 1m 37s | master passed | |
| +1 :green_heart: | spotless | 0m 50s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 55s | the patch passed | |
| +1 :green_heart: | compile | 3m 15s | the patch passed | |
| +1 :green_heart: | javac | 3m 15s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 57s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 40s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 11m 16s | Patch does not cause any errors with Hadoop 3.3.6 3.4.1. | |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | |
| 39m 2s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux ae3498d5fbc2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 8b0b6ff36b8ae61e093b582608dfbda71970c0a1 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/5/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 40s | master passed | |
| +1 :green_heart: | compile | 1m 6s | master passed | |
| +1 :green_heart: | javadoc | 0m 31s | master passed | |
| +1 :green_heart: | shadedjars | 6m 17s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 9s | the patch passed | |
| +1 :green_heart: | compile | 1m 0s | the patch passed | |
| +1 :green_heart: | javac | 1m 0s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 13s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 211m 43s | hbase-server in the patch passed. | |
| 239m 31s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 8f588827027a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 8b0b6ff36b8ae61e093b582608dfbda71970c0a1 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/5/testReport/ |
| Max. process+thread count | 4083 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/5/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
Hi @NihalJain, IMO TestDebugDumpRedaction tests the StateDumpServlet.getRedactedConfiguration() pretty thoroughly, so I did not move the method and add individual tests. Thanks for approving the PR. Let me know if you decide I should add individual tests for that as well. Otherwise, it looks like this PR can be merged.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 40s | master passed | |
| +1 :green_heart: | compile | 3m 29s | master passed | |
| +1 :green_heart: | checkstyle | 1m 0s | master passed | |
| +1 :green_heart: | spotbugs | 1m 41s | master passed | |
| +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 5s | the patch passed | |
| +1 :green_heart: | compile | 3m 27s | the patch passed | |
| +1 :green_heart: | javac | 3m 27s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 59s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 45s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 12m 29s | Patch does not cause any errors with Hadoop 3.3.6 3.4.1. | |
| +1 :green_heart: | spotless | 0m 46s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | |
| 42m 12s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 23f145aeb063 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1ad06f1f9b3aa2467feae2a9ebc3ae9450b324d1 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/6/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 16s | master passed | |
| +1 :green_heart: | compile | 1m 1s | master passed | |
| +1 :green_heart: | javadoc | 0m 29s | master passed | |
| +1 :green_heart: | shadedjars | 6m 13s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 5s | the patch passed | |
| +1 :green_heart: | compile | 1m 0s | the patch passed | |
| +1 :green_heart: | javac | 1m 0s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 13s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 212m 5s | hbase-server in the patch passed. | |
| 239m 23s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7545 |
| JIRA Issue | HBASE-29761 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux f6d3d36de758 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1ad06f1f9b3aa2467feae2a9ebc3ae9450b324d1 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/6/testReport/ |
| Max. process+thread count | 4134 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7545/6/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
Hi @NihalJain, IMO
TestDebugDumpRedactiontests theStateDumpServlet.getRedactedConfiguration()pretty thoroughly, so I did not move the method and add individual tests. Thanks for approving the PR. Let me know if you decide I should add individual tests for that as well. Otherwise, it looks like this PR can be merged.
Hey lgtm, we are good here
Thanks @kgeisz for this fix, I now merged this to master branch.
Can you please prepare a backport PR for branch-2?
Thanks @kgeisz for this fix, I now merged this to master branch. Can you please prepare a backport PR for branch-2?
@PDavid, no problem! Thanks for merging it. And yes, I will backport to branch-2. Should it go to branch-3 as well?
@PDavid, I spoke with @taklwu. I will backport to branch-2, branch-2.6, and branch-3.
@PDavid, I spoke with @taklwu. I will backport to branch-2, branch-2.6, and branch-3.
Great, many thanks @kgeisz! :+1: