hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-19077. Remove use of javax.ws.rs.core.HttpHeaders
Description of PR
HADOOP-19077 - part of moving to jersey 2 or 3 is to reduce our dependence on jersey 1
How was this patch tested?
CI build
For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
@steveloughran @pjfanning
Here's some good news to share: I successfully replaced Jersey 1 with Jersey 2 in the nodemanager module. The unit tests for nodemanager are passing, which is a positive step. I will now proceed with upgrading Jersey 2 in other modules. It seems that shading the package may not be necessary anymore; we can directly replace it. I'm hopeful that the upgrade process will proceed smoothly in other modules (at least I have a bit more confidence now).
Jersey 2 uses Jakarta-related packages, which include an upgrade from javax. My test results indicate that it's compatible. Therefore, I think we don't need to use this method for replacement. It's better to use the upgraded Jakarta packages instead of javax.
This PR can discuss the merge later. I will submit HADOOP-15984 within 1-2 weeks, and then we will discuss the javax upgrade together.
@slfan1989 this PR is more about what do we want to get HTTP header names from a jar that is causing us issues. Why not get them from another jar? I don't see why this change gets in the way of HADOOP-15984.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 43m 53s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 25s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 29s | trunk passed | |
| +1 :green_heart: | compile | 18m 10s | trunk passed | |
| +1 :green_heart: | checkstyle | 4m 55s | trunk passed | |
| +1 :green_heart: | mvnsite | 8m 36s | trunk passed | |
| +1 :green_heart: | javadoc | 7m 5s | trunk passed | |
| -1 :x: | spotbugs | 2m 54s | /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html | hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 11s | /branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 38m 14s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 5m 29s | the patch passed | |
| +1 :green_heart: | compile | 19m 19s | the patch passed | |
| +1 :green_heart: | javac | 19m 19s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 46s | the patch passed | |
| +1 :green_heart: | mvnsite | 8m 38s | the patch passed | |
| +1 :green_heart: | javadoc | 7m 3s | the patch passed | |
| +1 :green_heart: | spotbugs | 17m 45s | the patch passed | |
| +1 :green_heart: | shadedclient | 40m 54s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 19m 33s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 270m 23s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 108m 35s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | unit | 9m 2s | hadoop-mapreduce-client-app in the patch passed. | |
| +1 :green_heart: | unit | 0m 52s | hadoop-yarn-server-router in the patch passed. | |
| +1 :green_heart: | unit | 22m 6s | hadoop-yarn-services-core in the patch passed. | |
| +1 :green_heart: | unit | 2m 26s | hadoop-yarn-services-api in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 12s | The patch does not generate ASF License warnings. | |
| 737m 28s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.hdfs.server.datanode.TestDataNodeLifeline | |
| hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6554 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9a7963b43b94 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5ab18eb52c43bef0b9c3d496069d72dde417058b |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/2/testReport/ |
| Max. process+thread count | 2587 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/2/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 49s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 30s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 34s | trunk passed | |
| +1 :green_heart: | compile | 17m 36s | trunk passed | |
| +1 :green_heart: | checkstyle | 4m 50s | trunk passed | |
| +1 :green_heart: | mvnsite | 8m 44s | trunk passed | |
| +1 :green_heart: | javadoc | 7m 23s | trunk passed | |
| -1 :x: | spotbugs | 2m 47s | /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html | hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 10s | /branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 38m 57s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 5m 28s | the patch passed | |
| +1 :green_heart: | compile | 17m 36s | the patch passed | |
| +1 :green_heart: | javac | 17m 36s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 47s | the patch passed | |
| +1 :green_heart: | mvnsite | 8m 43s | the patch passed | |
| +1 :green_heart: | javadoc | 7m 1s | the patch passed | |
| +1 :green_heart: | spotbugs | 17m 3s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 13s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 19m 46s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 42s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 264m 9s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | unit | 108m 58s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | unit | 9m 3s | hadoop-mapreduce-client-app in the patch passed. | |
| +1 :green_heart: | unit | 0m 53s | hadoop-yarn-server-router in the patch passed. | |
| +1 :green_heart: | unit | 22m 1s | hadoop-yarn-services-core in the patch passed. | |
| +1 :green_heart: | unit | 3m 5s | hadoop-yarn-services-api in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 13s | The patch does not generate ASF License warnings. | |
| 685m 37s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.crypto.TestCryptoCodec |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6554 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1252669932f9 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5ab18eb52c43bef0b9c3d496069d72dde417058b |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/3/testReport/ |
| Max. process+thread count | 3113 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/3/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@pjfanning can you please rebase this, some issue with the build I believe, I triggered it twice, the test result link isn't working it is redirecting to the main build page rather than showing the test report https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/3/testReport/
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 18m 20s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 3s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 35s | trunk passed | |
| +1 :green_heart: | compile | 20m 50s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 17m 44s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 46s | trunk passed | |
| +1 :green_heart: | mvnsite | 8m 27s | trunk passed | |
| +1 :green_heart: | javadoc | 7m 1s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 7m 8s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| -1 :x: | spotbugs | 2m 47s | /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html | hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 8s | /branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 39m 13s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 5m 22s | the patch passed | |
| +1 :green_heart: | compile | 18m 18s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 18s | the patch passed | |
| +1 :green_heart: | compile | 17m 27s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 27s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 41s | the patch passed | |
| +1 :green_heart: | mvnsite | 8m 22s | the patch passed | |
| +1 :green_heart: | javadoc | 7m 3s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 7m 6s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 16m 33s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 8s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 15s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 42s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 262m 2s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | unit | 109m 7s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | unit | 8m 52s | hadoop-mapreduce-client-app in the patch passed. | |
| +1 :green_heart: | unit | 0m 51s | hadoop-yarn-server-router in the patch passed. | |
| +1 :green_heart: | unit | 21m 30s | hadoop-yarn-services-core in the patch passed. | |
| +1 :green_heart: | unit | 2m 25s | hadoop-yarn-services-api in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 751m 25s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6554 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e280e96b7f9f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b28a862dac6e6859c9d677be150d2edd07703a4a |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/4/testReport/ |
| Max. process+thread count | 2497 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6554/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
I don't think these spotbugs issues have been caused by this PR's changes.
@ayushtkn Thank you for reviewing the pr. This pr can be merged. @pjfanning Thanks for the contribution!
I have been quite busy lately, which caused a delay in addressing HADOOP-15984. I will have some time off in the next 2-3 days and will continue working on resolving HADOOP-15984.
I still have some concerns regarding the replacement of the javax package with another package. The javax package is a commonly used package, and I don't see how replacing it would solve any practical issues.
I still have a question, why there is a private package in the hadoop project. @pjfanning
<dependency>
<groupId>com.github.pjfanning</groupId>
<artifactId>jersey-json</artifactId>
</dependency>
@slfan1989 https://github.com/pjfanning/jersey-1.x provides some background. The dependency on my fork of jersey-json can be removed when the move to Jersey 2.x is done.