tez
tez copied to clipboard
apache
TEZ-4549. Upgrade Hadoop Version to 3.4.0.
JIRA: TEZ-4549. Upgrade Hadoop Version to 3.4.0.
hadoop has been upgraded to 3.4.0, try to upgrade hadoop to 3.4.0.
Local compilation successful.
[INFO] tez ................................................ SUCCESS [ 2.206 s]
[INFO] hadoop-shim ........................................ SUCCESS [ 27.433 s]
[INFO] tez-api ............................................ SUCCESS [ 36.299 s]
[INFO] tez-build-tools .................................... SUCCESS [ 0.233 s]
[INFO] tez-common ......................................... SUCCESS [ 2.093 s]
[INFO] tez-runtime-internals .............................. SUCCESS [ 8.123 s]
[INFO] tez-runtime-library ................................ SUCCESS [ 16.348 s]
[INFO] tez-mapreduce ...................................... SUCCESS [ 10.233 s]
[INFO] tez-examples ....................................... SUCCESS [ 1.769 s]
[INFO] tez-dag ............................................ SUCCESS [ 18.068 s]
[INFO] tez-tests .......................................... SUCCESS [ 18.191 s]
[INFO] tez-ext-service-tests .............................. SUCCESS [ 6.580 s]
[INFO] tez-ui ............................................. SUCCESS [ 46.579 s]
[INFO] tez-plugins ........................................ SUCCESS [ 0.469 s]
[INFO] tez-protobuf-history-plugin ........................ SUCCESS [ 11.401 s]
[INFO] tez-yarn-timeline-history .......................... SUCCESS [ 5.569 s]
[INFO] tez-yarn-timeline-history-with-acls ................ SUCCESS [ 4.206 s]
[INFO] tez-yarn-timeline-cache-plugin ..................... SUCCESS [ 37.988 s]
[INFO] tez-yarn-timeline-history-with-fs .................. SUCCESS [ 3.089 s]
[INFO] tez-history-parser ................................. SUCCESS [ 16.163 s]
[INFO] tez-aux-services ................................... SUCCESS [ 8.207 s]
[INFO] tez-tools .......................................... SUCCESS [ 0.103 s]
[INFO] tez-perf-analyzer .................................. SUCCESS [ 0.057 s]
[INFO] tez-job-analyzer ................................... SUCCESS [ 2.805 s]
[INFO] tez-javadoc-tools .................................. SUCCESS [ 0.905 s]
[INFO] hadoop-shim-impls .................................. SUCCESS [ 0.058 s]
[INFO] hadoop-shim-2.8 .................................... SUCCESS [ 1.576 s]
[INFO] tez-dist ........................................... SUCCESS [ 30.981 s]
[INFO] Tez ................................................ SUCCESS [ 0.287 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:18 min
[INFO] Finished at: 2024-04-03T22:58:57+08:00
[INFO] ------------------------------------------------------------------------
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| _ Prechecks _ | |||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| _ master Compile Tests _ | |||
| +1 :green_heart: | mvninstall | 18m 7s | master passed |
| +1 :green_heart: | compile | 2m 28s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 |
| +1 :green_heart: | compile | 2m 13s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 |
| +1 :green_heart: | javadoc | 1m 50s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 |
| +1 :green_heart: | javadoc | 1m 11s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 |
| _ Patch Compile Tests _ | |||
| +1 :green_heart: | mvninstall | 6m 11s | the patch passed |
| +1 :green_heart: | compile | 2m 34s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 |
| +1 :green_heart: | javac | 2m 34s | the patch passed |
| +1 :green_heart: | compile | 2m 18s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 |
| +1 :green_heart: | javac | 2m 18s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 15s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 |
| +1 :green_heart: | javadoc | 1m 10s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 |
| _ Other Tests _ | |||
| -1 :x: | unit | 3m 53s | root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| 45m 48s |
| Reason | Tests |
|---|---|
| Failed junit tests | tez.common.TestTezCommonUtils |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/342 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile |
| uname | Linux 1ec75fbf2bd4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / f080031f5 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 |
| unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/artifact/out/patch-unit-root.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/testReport/ |
| Max. process+thread count | 96 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/console |
| versions | git=2.34.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
We usually couple Hadoop upgrade on Tez & Hive together, they have a history of not working when not on same version, everytime we didn't, it creates classpath issues when running queries in hive, we had to quickly push the last Tez release to unblock hive, since hive with 3.3.6 wasn't compatible with Tez on hadoop-3.3.1, some Htrace stuff...
Hive doesn't compile as of now with Hadoop-3.4.0, so, I think we should hold this till Hive gets that sorted
Thank you for explanation!
From changes POV, Hadoop 3.4.0 has zookeeper version 3.8.3 which uses logback. We have to explicitly exclude it from hadoop dependecies, otherwise it will lead to class loader issue. For example, in hive, If logback jar gets picked up first then hive-log4j2.properties won't be honored. Same is mentioned in description of https://github.com/apache/hadoop/pull/6582
From changes POV, Hadoop 3.4.0 has zookeeper version 3.8.3 which uses logback. We have to explicitly exclude it from hadoop dependecies, otherwise it will lead to class loader issue. For example, in hive, If logback jar gets picked up first then hive-log4j2.properties won't be honored. Same is mentioned in description of apache/hadoop#6582
@Aggarwal-Raghav We can trying to upgrading Hadoop version to 3.4.0 in Hive https://github.com/apache/hive/pull/5500. Any thought about refine this change https://github.com/apache/hive/pull/5500? Thanks.
Have added comments on hive PR https://github.com/apache/hive/pull/5500