hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-18197. Upgrade protobuf to 3.21.7 (through upgraded hadoop-shaded-protobuf jar)
Description of PR
This patch bumps up the protobuf version so that Hadoop is not a vulnerable to CVE-2021-22569.
Depends on a version of hadoop-shaded-protobuf_3_7 with the update; this PR does this by depending on 1.2.0-SNAPSHOT...this is only going to work for local builds
How was this patch tested?
non-native build on a local mac against a local build of the thirdparty jar.
none of the docker changes/build instructions have been tested yet
For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [X] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 47m 45s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | shellcheck | 0m 0s | Shellcheck was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +0 :ok: | hadolint | 0m 0s | hadolint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 25s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 26s | trunk passed | |
| +1 :green_heart: | compile | 21m 42s | trunk passed | |
| +1 :green_heart: | mvnsite | 20m 9s | trunk passed | |
| +1 :green_heart: | javadoc | 7m 54s | trunk passed | |
| +1 :green_heart: | shadedclient | 29m 33s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 39s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 1m 11s | /patch-mvninstall-root.txt | root in the patch failed. |
| -1 :x: | compile | 1m 1s | /patch-compile-root.txt | root in the patch failed. |
| -1 :x: | javac | 1m 1s | /patch-compile-root.txt | root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -1 :x: | mvnsite | 0m 51s | /patch-mvnsite-root.txt | root in the patch failed. |
| +1 :green_heart: | xmllint | 0m 0s | No new issues. | |
| -1 :x: | javadoc | 7m 29s | /results-javadoc-javadoc-root.txt | root generated 516 new + 2329 unchanged - 0 fixed = 2845 total (was 2329) |
| -1 :x: | shadedclient | 9m 58s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 8m 7s | /patch-unit-root.txt | root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 9s | The patch does not generate ASF License warnings. | |
| 187m 45s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4418 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs hadolint mvnsite unit compile javac javadoc mvninstall shadedclient xmllint |
| uname | Linux 3c012a890e06 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 92455608e48f2a1af9887648c06fe22135fe3974 |
| Default Java | Red Hat, Inc.-1.8.0_332-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/1/testReport/ |
| Max. process+thread count | 595 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/1/console |
| versions | git=2.9.5 maven=3.6.3 xmllint=20901 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 37m 52s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | shellcheck | 0m 0s | Shellcheck was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +0 :ok: | hadolint | 0m 0s | hadolint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 37s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 27m 5s | trunk passed | |
| +1 :green_heart: | compile | 21m 9s | trunk passed | |
| +1 :green_heart: | mvnsite | 19m 46s | trunk passed | |
| +1 :green_heart: | javadoc | 7m 52s | trunk passed | |
| +1 :green_heart: | shadedclient | 26m 34s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 0m 48s | /patch-mvninstall-root.txt | root in the patch failed. |
| -1 :x: | mvninstall | 0m 20s | /patch-mvninstall-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 22s | /patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt | hadoop-yarn-api in the patch failed. |
| -1 :x: | compile | 0m 40s | /patch-compile-root.txt | root in the patch failed. |
| -1 :x: | javac | 0m 40s | /patch-compile-root.txt | root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -1 :x: | mvnsite | 0m 31s | /patch-mvnsite-root.txt | root in the patch failed. |
| +1 :green_heart: | xmllint | 0m 0s | No new issues. | |
| -1 :x: | javadoc | 0m 33s | /patch-javadoc-root.txt | root in the patch failed. |
| -1 :x: | shadedclient | 2m 9s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 6m 48s | /patch-unit-root.txt | root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | |
| 164m 9s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4418 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs hadolint mvnsite unit compile javac javadoc mvninstall shadedclient xmllint |
| uname | Linux 67905c1ae8ab 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d272048600695c0005e2fcd4dd22aa6449393c1a |
| Default Java | Red Hat, Inc.-1.8.0_345-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/2/testReport/ |
| Max. process+thread count | 624 (vs. ulimit of 5500) |
| modules | C: hadoop-project . hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/2/console |
| versions | git=2.9.5 maven=3.6.3 xmllint=20901 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 46m 14s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | shellcheck | 0m 0s | Shellcheck was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +0 :ok: | hadolint | 0m 0s | hadolint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 45s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 29m 26s | trunk passed | |
| +1 :green_heart: | compile | 22m 17s | trunk passed | |
| +1 :green_heart: | mvnsite | 20m 45s | trunk passed | |
| +1 :green_heart: | javadoc | 7m 57s | trunk passed | |
| +1 :green_heart: | shadedclient | 30m 33s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 1m 11s | /patch-mvninstall-root.txt | root in the patch failed. |
| -1 :x: | compile | 0m 59s | /patch-compile-root.txt | root in the patch failed. |
| -1 :x: | javac | 0m 59s | /patch-compile-root.txt | root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -1 :x: | mvnsite | 0m 48s | /patch-mvnsite-root.txt | root in the patch failed. |
| +1 :green_heart: | xmllint | 0m 0s | No new issues. | |
| -1 :x: | javadoc | 7m 33s | /results-javadoc-javadoc-root.txt | root generated 534 new + 2269 unchanged - 0 fixed = 2803 total (was 2269) |
| -1 :x: | shadedclient | 9m 53s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 7m 42s | /patch-unit-root.txt | root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | |
| 189m 56s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4418 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs hadolint mvnsite unit compile javac javadoc mvninstall shadedclient xmllint |
| uname | Linux eeeb6886f515 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4f05bf48185e1cb3edce862286a3fc01b41ea451 |
| Default Java | Red Hat, Inc.-1.8.0_345-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/3/testReport/ |
| Max. process+thread count | 530 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4418/3/console |
| versions | git=2.9.5 maven=3.6.3 xmllint=20901 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@xizhu-mstr @tooptoop4 i'm not actively working on this; too many pressing issues and after getting 3.3.5 out the door I'm catching up with the internal stuff. Either of you two want to take it on?
I'd also like to get #4996 in; if anyone wants to run with that, I'd be very happy. We shouldn't need protobuf 2.5 on the CP given we aren't using it
@steveloughran - is this a genuine CVE in hadoop-shaded-protobuf or is it just to please the scanner gods? :)
usual ongoing protobuf issues; AFAIK none of them lethal. YMMV
I do want #4996 in so we can get protobuf 2.5 off the classpath. if you could take that up, it'd be good. that PR doesn't cut it, only make it optional. a followup would cut it.
@steveloughran The title states upgrade protobuf to 3.21.7 while the version downloaded is 3.21.1. hope you can bump it up whenever you'll get back to this.
says 3.21.x... we should take the latest one we can which doesn't include other surprises...pr and jira can be set to the final version which goes in as it is merged