apache
HBASE-27931 Update hadoop.version from 3.3.5 to 3.3.6
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 4m 30s | master passed | |
| +1 :green_heart: | compile | 10m 55s | master passed | |
| +1 :green_heart: | spotless | 1m 11s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 4m 34s | the patch passed | |
| +1 :green_heart: | compile | 12m 24s | the patch passed | |
| +1 :green_heart: | javac | 12m 24s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | xmllint | 0m 0s | No new issues. | |
| +1 :green_heart: | hadoopcheck | 15m 36s | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. | |
| +1 :green_heart: | spotless | 0m 54s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | |
| 60m 12s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6232 |
| Optional Tests | dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile |
| uname | Linux c8f983a40ea2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 42d0484cd82e8069ea634023672dca03b36e0504 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 189 (vs. ulimit of 30000) |
| modules | C: . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/console |
| versions | git=2.34.1 maven=3.9.8 xmllint=20913 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 5s | master passed | |
| +1 :green_heart: | compile | 2m 10s | master passed | |
| +1 :green_heart: | javadoc | 2m 6s | master passed | |
| +1 :green_heart: | shadedjars | 5m 16s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 59s | the patch passed | |
| +1 :green_heart: | compile | 2m 10s | the patch passed | |
| +1 :green_heart: | javac | 2m 10s | the patch passed | |
| +1 :green_heart: | javadoc | 2m 4s | the patch passed | |
| +1 :green_heart: | shadedjars | 5m 16s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 300m 10s | root in the patch passed. | |
| 332m 46s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6232 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d7e2b20c2342 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 42d0484cd82e8069ea634023672dca03b36e0504 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/testReport/ |
| Max. process+thread count | 8612 (vs. ulimit of 30000) |
| modules | C: . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
Maybe I missed some discussions on the dev list. Why we want to bump the default hadoop version to 3.3.6? We do not want to support 3.3.5 since it has new CVEs?
Yes, I started a discuss thread on the topic a few days ago.
I will reply there to keep the discussion in one place.
You'll need to make changes in several more places. Search the repo on github for "3.3.5" and you'll get a sense for this:
- the Jenkinsfiles download and cache specific hadoop build versions
- the hbase_personality.sh has a
hadoopcheckfunction that builds against myriad hadoop versions - the book includes a support matrix, which needs to be updated (on master only) according to these changes.
Thank you @ndimiduk . Much of that work has already been done. The docs refers to 3.3.5+, which should already cover 3.3.6, and 3.3.6 is already tested in HADOOPCHECK.
I have replaced 3.3.6 with 3.3.5 in QUICK_HADOOPCHECK, as 3.3.6 should already be tested as the default. I have also updated the cached version you pointed out.
(!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console in case of problems.
(!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console in case of problems.
Added a rather rambling summary of my thoughts to the discuss thread, @Apache9 : https://lists.apache.org/thread/h53ltygosg2gqo0ftdvqngf6h9z6brb1
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 55s | master passed | |
| +1 :green_heart: | compile | 9m 14s | master passed | |
| +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 8s | the patch passed | |
| +1 :green_heart: | compile | 7m 55s | the patch passed | |
| +1 :green_heart: | javac | 7m 55s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | shellcheck | 0m 1s | No new issues. | |
| +1 :green_heart: | xmllint | 0m 0s | No new issues. | |
| +1 :green_heart: | hadoopcheck | 11m 34s | Patch does not cause any errors with Hadoop 3.3.5 3.4.0. | |
| +1 :green_heart: | spotless | 0m 44s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | |
| 45m 4s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6232 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac xmllint hadoopcheck compile |
| uname | Linux d93118e90735 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e8a70d9de2646da82ff483450d99584025a88e0e |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 192 (vs. ulimit of 30000) |
| modules | C: . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console |
| versions | git=2.34.1 maven=3.9.8 shellcheck=0.8.0 xmllint=20913 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 3m 1s | master passed | |
| +1 :green_heart: | compile | 2m 2s | master passed | |
| +1 :green_heart: | javadoc | 1m 53s | master passed | |
| +1 :green_heart: | shadedjars | 5m 33s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 2m 51s | the patch passed | |
| +1 :green_heart: | compile | 2m 4s | the patch passed | |
| +1 :green_heart: | javac | 2m 4s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 52s | the patch passed | |
| +1 :green_heart: | shadedjars | 5m 28s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 289m 11s | root in the patch passed. | |
| 320m 58s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6232 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8b037ce5be12 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e8a70d9de2646da82ff483450d99584025a88e0e |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/testReport/ |
| Max. process+thread count | 8913 (vs. ulimit of 30000) |
| modules | C: . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
You've resolved my comments, but Duo's post to the mailing list thread on 11 Sept is a valid point about insufficient verification with existing hadoop checks. We need to either run a reasonably thorough test suite against the older versions or we need to explicitly drop support for them. Let's continue discussing on the mailing list.
Thanks Nick. For me, I think we could at least extend the current client integration tests, to run multiple combinations for client and server versions. We could continue the discussion on the dev mailing list.