hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard

Commit to remove vulnerabilities raised for: HADOOP-19074

Open prathapsagar opened this issue 1 year ago • 4 comments

  1. Update Gauva to Version: 32.0.1-jre
  2. Update Protobuf to Version: 3.21.12
  3. Update Avro to Version: 1.11.3
  4. Updated private access in the below files for Avro compability:
  • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/TestAvroSerialization.java
  • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobQueueChangeEvent.java
  • hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java
  • hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
  • hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java
  • hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java

Description of PR

How was this patch tested?

For code changes:

  • [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

prathapsagar avatar Feb 25 '24 14:02 prathapsagar

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 48s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+0 :ok: xmllint 0m 1s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 :ok: mvndep 35m 42s Maven dependency ordering for branch
+1 :green_heart: mvninstall 41m 11s trunk passed
+1 :green_heart: compile 19m 54s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 17m 57s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 4m 57s trunk passed
+1 :green_heart: mvnsite 4m 1s trunk passed
+1 :green_heart: javadoc 3m 23s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 2m 41s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+0 :ok: spotbugs 0m 41s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
-1 :x: spotbugs 2m 31s /branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings.
+1 :green_heart: shadedclient 38m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 53s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 8s the patch passed
+1 :green_heart: compile 18m 50s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 18m 50s the patch passed
+1 :green_heart: compile 17m 41s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 17m 41s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 4m 56s /results-checkstyle-root.txt root: The patch generated 1 new + 89 unchanged - 0 fixed = 90 total (was 89)
+1 :green_heart: mvnsite 3m 58s the patch passed
+1 :green_heart: javadoc 3m 12s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 2m 49s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+0 :ok: spotbugs 0m 36s hadoop-project has no data from spotbugs
-1 :x: shadedclient 2m 44s patch has errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 0m 35s hadoop-project in the patch passed.
+1 :green_heart: unit 20m 29s hadoop-common in the patch passed.
+1 :green_heart: unit 7m 32s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: unit 0m 46s hadoop-rumen in the patch passed.
+1 :green_heart: asflicense 1m 4s The patch does not generate ASF License warnings.
276m 38s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6586
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname Linux eaae40a63a9a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 5b8e2af9ac050bb814e67c2ed6c642d34c2986dc
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/testReport/
Max. process+thread count 1257 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-rumen U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Feb 25 '24 19:02 hadoop-yetus

Hi @dineshchitlangia I have resolved the check style violation and raised a new PR with the changes please validate: https://github.com/apache/hadoop/pull/6586

prathapsagar avatar Feb 29 '24 07:02 prathapsagar

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 17m 23s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: xmllint 0m 0s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 :ok: mvndep 14m 29s Maven dependency ordering for branch
+1 :green_heart: mvninstall 37m 3s trunk passed
+1 :green_heart: compile 20m 16s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 18m 26s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 4m 50s trunk passed
+1 :green_heart: mvnsite 4m 1s trunk passed
+1 :green_heart: javadoc 3m 17s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 2m 49s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+0 :ok: spotbugs 0m 41s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
-1 :x: spotbugs 2m 31s /branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings.
+1 :green_heart: shadedclient 38m 38s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 41s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 7s the patch passed
+1 :green_heart: compile 18m 47s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 18m 47s the patch passed
+1 :green_heart: compile 18m 8s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 18m 8s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 4m 50s the patch passed
+1 :green_heart: mvnsite 4m 1s the patch passed
+1 :green_heart: javadoc 3m 13s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 2m 50s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+0 :ok: spotbugs 0m 36s hadoop-project has no data from spotbugs
-1 :x: shadedclient 2m 44s patch has errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 0m 36s hadoop-project in the patch passed.
+1 :green_heart: unit 20m 27s hadoop-common in the patch passed.
+1 :green_heart: unit 7m 30s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: unit 0m 47s hadoop-rumen in the patch passed.
+1 :green_heart: asflicense 1m 4s The patch does not generate ASF License warnings.
268m 44s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/2/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6586
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname Linux 283fc36b66d6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2d96b0cdcb4968a31dc681582b9401083c2b6d2e
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/2/testReport/
Max. process+thread count 1256 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-rumen U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Feb 29 '24 11:02 hadoop-yetus

@prathapsagar thanks for starting this. you are about to discover why it so hard to update dependencies.

Everything we want to use in our own code should be defined in the thirdparty module, https://github.com/apache/hadoop-thirdparty

the version numbers of things like protobuf in our own code must match those of the library version we build with, but they aren't where the values are incremented. (note: we should add comments there).

steveloughran avatar Mar 01 '24 13:03 steveloughran

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
-1 :x: patch 0m 54s https://github.com/apache/hadoop/pull/6586 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hadoop/pull/6586
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6586/1/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 24 '24 20:04 hadoop-yetus