TEZ-4420: Upgrade to Hadoop 3.3.4

[abstractdog]

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	10m 52s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 :green_heart:	mvninstall	18m 26s	master passed
+1 :green_heart:	compile	3m 23s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	compile	3m 1s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	3m 25s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	2m 23s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+1 :green_heart:	mvninstall	7m 2s	the patch passed
+1 :green_heart:	compile	3m 14s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javac	3m 14s	the patch passed
+1 :green_heart:	compile	2m 53s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	2m 53s	the patch passed
+1 :green_heart:	whitespace	0m 1s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 3s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 52s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	2m 19s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Other Tests _
-1 :x:	unit	1m 0s	root in the patch failed.
+0 :ok:	asflicense	0m 45s	ASF License check generated no output?
		62m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux d80a98b53d0c 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 24a77c93f
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/1/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/1/testReport/
Max. process+thread count	79 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	1m 2s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 :green_heart:	mvninstall	15m 48s	master passed
+1 :green_heart:	compile	2m 33s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	compile	2m 20s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	2m 33s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	1m 50s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+1 :green_heart:	mvninstall	5m 31s	the patch passed
+1 :green_heart:	compile	2m 33s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javac	2m 33s	the patch passed
+1 :green_heart:	compile	2m 22s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	2m 22s	the patch passed
+1 :green_heart:	whitespace	0m 0s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 1s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 11s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	1m 49s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Other Tests _
-1 :x:	unit	4m 12s	root in the patch failed.
+1 :green_heart:	asflicense	0m 39s	The patch does not generate ASF License warnings.
		46m 26s

Reason	Tests
Failed junit tests	tez.common.TestTezCommonUtils

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux cb9fead591f0 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 24a77c93f
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/2/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/2/testReport/
Max. process+thread count	295 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	0m 59s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 :green_heart:	mvninstall	16m 18s	master passed
+1 :green_heart:	compile	2m 34s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	compile	2m 24s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	2m 35s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	1m 49s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+1 :green_heart:	mvninstall	5m 30s	the patch passed
+1 :green_heart:	compile	2m 35s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javac	2m 35s	the patch passed
+1 :green_heart:	compile	2m 20s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	2m 20s	the patch passed
+1 :green_heart:	whitespace	0m 0s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 1s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 13s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	1m 50s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Other Tests _
-1 :x:	unit	18m 0s	root in the patch failed.
+1 :green_heart:	asflicense	0m 41s	The patch does not generate ASF License warnings.
		60m 52s

Reason	Tests
Failed junit tests	tez.test.TestDAGRecovery2
	tez.test.TestExceptionPropagation
	tez.test.TestPipelinedShuffle
	tez.test.TestSecureShuffle
	tez.test.TestTezJobs
	tez.test.TestLocalMode
	tez.test.TestDAGRecovery
	tez.test.TestAMRecovery
	tez.test.TestRecovery
	tez.test.TestFaultTolerance
	tez.test.TestAM
	tez.mapreduce.TestMRRJobsDAGApi
	tez.mapreduce.TestMRRJobs

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/3/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux c53d2b3a840a 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / e5a557802
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/3/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/3/testReport/
Max. process+thread count	1089 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	16m 56s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 :ok:	mvndep	6m 55s	Maven dependency ordering for branch
+1 :green_heart:	mvninstall	10m 23s	master passed
+1 :green_heart:	compile	3m 15s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	compile	3m 3s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	3m 8s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	2m 27s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+0 :ok:	mvndep	1m 20s	Maven dependency ordering for patch
+1 :green_heart:	mvninstall	5m 0s	the patch passed
+1 :green_heart:	compile	3m 22s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javac	3m 22s	the patch passed
+1 :green_heart:	compile	3m 4s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	3m 4s	the patch passed
+1 :green_heart:	whitespace	0m 0s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 3s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 47s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 :green_heart:	javadoc	2m 21s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Other Tests _
+1 :green_heart:	unit	38m 7s	tez-tests in the patch passed.
-1 :x:	unit	71m 24s	root in the patch failed.
+1 :green_heart:	asflicense	1m 19s	The patch does not generate ASF License warnings.
		176m 2s

Reason	Tests
Failed junit tests	tez.history.TestHistoryParser
	tez.dag.history.ats.acls.TestATSHistoryWithACLs

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/4/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 441eefcdb92a 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / e5a557802
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/4/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/4/testReport/
Max. process+thread count	2089 (vs. ulimit of 5500)
modules	C: tez-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/4/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[pjfanning]

The test issues seem to be related to mockito

java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer at org.apache.tez.test.TestAM.setup(TestAM.java:66) Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer at org.apache.tez.test.TestAM.setup(TestAM.java:66)

[ayushtkn]

You are checking the wrong build result, mockito also was a problem, but we upgraded it and the build result got sorted. Check the result here of this test in the latest build: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/4/testReport/org.apache.tez.history/TestHistoryParser/testParserWithFailedJob/

There is an exception:

java.lang.AbstractMethodError: javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
	at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:96)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:911)

[steveloughran]

hadoop 3.3.4 cuts jax.rs from the dependency graph

[zhangbutao]

@abstractdog Can we upgrade hadoop to 3.3.4? I have tested hive maser with hadoop3.3.4(https://github.com/apache/hive/pull/3578) , and if i change tez's hadoop dependency to 3.3.4, everything looks ok.

[abstractdog]

yes, let me check this PR and have some test coverage

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	36m 24s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 :ok:	mvndep	5m 15s	Maven dependency ordering for branch
+1 :green_heart:	mvninstall	10m 38s	master passed
+1 :green_heart:	compile	3m 11s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	2m 54s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	2m 58s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 24s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+0 :ok:	mvndep	1m 15s	Maven dependency ordering for patch
+1 :green_heart:	mvninstall	4m 41s	the patch passed
+1 :green_heart:	compile	3m 9s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	3m 9s	the patch passed
+1 :green_heart:	compile	2m 54s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	2m 54s	the patch passed
+1 :green_heart:	whitespace	0m 0s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 2s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 48s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 24s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
		_ Other Tests _
+1 :green_heart:	unit	40m 41s	tez-tests in the patch passed.
+1 :green_heart:	unit	67m 57s	root in the patch passed.
+1 :green_heart:	asflicense	1m 14s	The patch does not generate ASF License warnings.
		192m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/5/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 241629ee6fdf 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 55b6031fa
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/5/testReport/
Max. process+thread count	1384 (vs. ulimit of 5500)
modules	C: tez-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/5/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[zhangbutao]

Great! If this PR is merged into Tez 0.10.2, then we can continue to upgrade hadoop version in hive repo.

[pjfanning]

Can this be merged to unblock other upgrades?

[abstractdog]

Can this be merged to unblock other upgrades?

let me take a look soon what upgrade is this blocking at the moment? actually, hive is not blocked now AFAIK

[pjfanning]

It was Hive that I thought was blocked. It would still be nice to get this moved on but I guess if it's not blocking anything then it's not as urgent.

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	37m 14s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 :ok:	mvndep	6m 16s	Maven dependency ordering for branch
+1 :green_heart:	mvninstall	10m 6s	master passed
+1 :green_heart:	compile	3m 8s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	2m 53s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	2m 55s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 24s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+0 :ok:	mvndep	1m 17s	Maven dependency ordering for patch
+1 :green_heart:	mvninstall	4m 37s	the patch passed
+1 :green_heart:	compile	3m 5s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	3m 5s	the patch passed
+1 :green_heart:	compile	2m 53s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	2m 53s	the patch passed
+1 :green_heart:	whitespace	0m 0s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 2s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 49s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 24s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
		_ Other Tests _
-1 :x:	unit	41m 41s	tez-tests in the patch passed.
-1 :x:	unit	55m 35s	root in the patch failed.
+1 :green_heart:	asflicense	1m 11s	The patch does not generate ASF License warnings.
		181m 42s

Reason	Tests
Failed junit tests	tez.test.TestRecovery
	tez.test.TestRecovery

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/6/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 817c62819f47 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 40e9e0f6a
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/6/artifact/out/patch-unit-tez-tests.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/6/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/6/testReport/
Max. process+thread count	2090 (vs. ulimit of 5500)
modules	C: tez-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/6/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	1m 0s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 :ok:	mvndep	6m 24s	Maven dependency ordering for branch
+1 :green_heart:	mvninstall	10m 6s	master passed
+1 :green_heart:	compile	3m 5s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	2m 50s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javadoc	2m 55s	master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 24s	master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+0 :ok:	mvndep	1m 16s	Maven dependency ordering for patch
+1 :green_heart:	mvninstall	4m 42s	the patch passed
+1 :green_heart:	compile	3m 8s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	3m 8s	the patch passed
+1 :green_heart:	compile	2m 51s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	javac	2m 51s	the patch passed
+1 :green_heart:	whitespace	0m 1s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 3s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	2m 46s	the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 25s	the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
		_ Other Tests _
-1 :x:	unit	39m 11s	tez-tests in the patch passed.
-1 :x:	unit	51m 10s	root in the patch failed.
+1 :green_heart:	asflicense	1m 10s	The patch does not generate ASF License warnings.
		138m 35s

Reason	Tests
Failed junit tests	tez.test.TestRecovery
	tez.test.TestRecovery

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/7/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 85cf2879f424 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / ddb53cb01
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/7/artifact/out/patch-unit-tez-tests.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/7/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/7/testReport/
Max. process+thread count	2089 (vs. ulimit of 5500)
modules	C: tez-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/7/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

we are nearing code freeze for 3.3.5 btw; if there is some final blocker to upgrade, now is the time to identify it

[amanraj2520]

Hi @ayushtkn @abstractdog @pjfanning @steveloughran

Since this PR has been stale for a month, just following up to check whether I can help in validating any other test scenarios for the upgrade. Or if not, are we good to merge this PR ?

[amanraj2520]

Hi @ayushtkn @abstractdog @pjfanning @steveloughran,

I am driving the hive-3.2.0 release in Open Source for which we have decided to upgrade to hadoop 3.3.4. But since Tez 0.10.2 is using hadoop-3.3.1 there can be some integration issues for this stack. I know that we are currently discussing hadoop-3.3.5 release but it would take at least a couple of months to get a stable release candidate for the same which would also require a lot of testing.

This will be a blocker for the 3.2.0 release and since we have started to plan our tasks around this release, I was proposing to make a Tez release 0.10.3 where we can have this hadoop version upgraded to 3.3.4. This would insure that we can start working on the 3.2.0 without any blockers.

Please share your thoughts on the same. I am open to discussing this release further.

Thanks, Aman.

[ayushtkn]

Hi @amanraj2520 Just curious: What specifically you are targeting from Hadoop-3.3.4, mostly there would be some dependency upgrades in 3.3.4 & they must be in sync with the versions in current hive master.

Anyway if we plan to do so:

• We would need to upgrade the hadoop version first in master & then only we should bring that back to 3.2.0. Any upgrade or fix should go to hive master first and then only get backported. So, that is some additional work for you.

• Last time the plan was to move to hadoop-3.3.3, but Tez got issues due to some netty version conflict, and that issue wasn't getting flagged in the Hive unit tests, but only in actual cluster with Tez, so we should 'confirm' running basic tests with Hive before we push for a Tez release, Getting to know "There are issues with this version of Hadoop in Hive" once we have a Tez release, won't be a good for us, we won't be able to immediately push for a new Hadoop release with the fix, nor a Tez release with the fixed Hadoop release version very quickly.

BTW. AFAIK Hadoop-3.3.5 is just awaiting one ABFS related jira fix and should be out in a month's time(if no new blockers), We can ask @steveloughran if we want to push that quickly for Hive. Rest, I am ok with whatever ways you folks feel good, @abstractdog can help share pointers around the timelines for a new Tez release, If we move with 3.3.4 then I feel moving to 3.3.5 won't be happening any time soon, Getting the last Tez release itself was very tough for us.

[amanraj2520]

Hi @ayushtkn Thanks a lot for looking into this I checked the PR you were involved about upgrading Hive to 3.3.4 so I am referring this PR : https://github.com/apache/hive/pull/3578

I have answered your questions below: The reason why we wanted to use Hadoop 3.3.4 was because of some component vulnerabilities issues which were fixed by version upgrades from 3.3.1. Also 3.3.4 is the latest stable release version of Hadoop so it would be better to upgrade to the latest in Hive as well. Having said that, I see in the PR of Hive I have referred, testing had started for Hadoop 3.3.4 with Tez 0.10.2 and the possible solution to them was to upgrade Hive and Tez both to 3.3.4 simultaneously. Therefore, we want to start fresh and include the latest Hadoop 3.3.4 and Tez 0.10.3 (if approved will use 3.3.4), so that we ensure both hive-3.2.0 release and oss master start to use the same version. I can take up the responsibility of driving this 0.10.3 Tez release if approved. I don't think major changes will go for this Tez release. We can just upgrade the Hadoop to 3.3.4 on top of 0.10.2.

1. I completely agree with you on this. I think the PR was raised for upgrade in master was successful. So, I think we just need to upgrade the tez version on top of this PR and I can test it in my clusters before signing it off.
2. I see that Hive with 3.3.4 Hadoop and 0.10.2 passed all the unit test cases right? By basic tests you mean unit tests right? It was only while testing on the cluster we found some issues since Tez classpath had 3.3.1 jars instead of 3.3.4. So I think if we just upgrade to Hadoop 3.3.4 in Tez and use that Tez version in Hive, we should be good.

Please let me know if you have any concerns, Ayush. Looking forward to your reply. @abstractdog Please suggest anything you find concerning in this release. Happy to help.

Thanks, Aman.

[ayushtkn]

I was talking about https://github.com/apache/hive/pull/3279 here we tried till 3.3.3 and all the unit tests were passing but on actual cluster things weren't working. So, I meant apart from the unit tests, we need to test on actual clusters as well.

The failure was flagged here: https://github.com/apache/hive/pull/3279#issuecomment-1141893046

[amanraj2520]

Hi @ayushtkn

I am familiar with this error. This happens because Tez 0.10.1 is using Hadoop 3.1.3 which in turn uses jetty 9.3* which does not have this method but when we upgrade to Hadoop 3.3.4 it uses this method. So basically I am saying that if we maintained the same version of Hadoop in Hive and Tez, this would not have been the issue. Having said this, I will make sure that the 0.10.3 is validated on clusters including unit tests Ayush.

[steveloughran]

3.3.4 is the one where we fixed the tez incompatibilities for you 🙂

i plan to cut the 3.3.5 rc0 this week but it's a test of the rc process (x86 and arm) rather than something we intend to bring to a vote.

it'd be good to test hive and tez with the rc to see if we have caused any regressions, as now will be the time to fix. there's been a lot of dependency updates to stop CVEs, but we've left alone the ones we know cause problems downstream (jackson is on a cve-fixed 2.12.x release)

[amanraj2520]

@steveloughran So do you think having a new Tez release with 3.3.4 should help or not?

[amanraj2520]

@abstractdog @steveloughran Can you please look into this?

[steveloughran]

3.3.4 should be good; if not try with a 3.3.5 snapshot (build yourself) and see if that fixes things.

i'd suggest going with 3.3.4 if it works, so the 3.3.5 release isn't a blocker. you can do an upgrade after

[abstractdog]

I believe from tez->hadoop point of view, we usually rely only on precommit testing (including lots of minicluster tests) if something is broken from hive -> tez -> hadoop, we should report it separately I guess we should simply go on with hadoop 3.3.4 now, and resolve this ticket

TestRecovery passed locally, I'm restarting precommit tests

can someone approve this PR? @rbalamohan , @jteagles

[tez-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Comment
+0 :ok:	reexec	36m 14s	Docker mode activated.
		_ Prechecks _
+1 :green_heart:	dupname	0m 0s	No case conflicting files found.
+1 :green_heart:	@author	0m 0s	The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 :ok:	mvndep	6m 32s	Maven dependency ordering for branch
+1 :green_heart:	mvninstall	10m 4s	master passed
+1 :green_heart:	compile	3m 12s	master passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 :green_heart:	compile	3m 2s	master passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 :green_heart:	javadoc	3m 9s	master passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 :green_heart:	javadoc	2m 26s	master passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
		_ Patch Compile Tests _
+0 :ok:	mvndep	1m 17s	Maven dependency ordering for patch
+1 :green_heart:	mvninstall	6m 9s	the patch passed
+1 :green_heart:	compile	4m 7s	the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 :green_heart:	javac	4m 7s	the patch passed
+1 :green_heart:	compile	3m 42s	the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 :green_heart:	javac	3m 42s	the patch passed
+1 :green_heart:	whitespace	0m 0s	The patch has no whitespace issues.
+1 :green_heart:	xml	0m 2s	The patch has no ill-formed XML file.
+1 :green_heart:	javadoc	3m 24s	the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 :green_heart:	javadoc	2m 53s	the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
		_ Other Tests _
+1 :green_heart:	unit	40m 17s	tez-tests in the patch passed.
+1 :green_heart:	unit	72m 24s	root in the patch passed.
+1 :green_heart:	asflicense	1m 14s	The patch does not generate ASF License warnings.
		201m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/8/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/tez/pull/213
JIRA Issue	TEZ-4420
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux e4562a85b404 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 2fd7df4cf
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/8/testReport/
Max. process+thread count	1388 (vs. ulimit of 5500)
modules	C: tez-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/8/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[amanraj2520]

@rbalamohan @jteagles Can you please approve this PR. Also @abstractdog are we going with the tez 0.10.3 release which will have hadoop 3.3.4.