tez icon indicating copy to clipboard operation
tez copied to clipboard
verified publisher icon apache

TEZ-4654: Migrate from commons-lang2.x to commons-lang3.x

Open Aggarwal-Raghav opened this issue 3 months ago • 7 comments

  • Upgrade to commons-lang-3.19.0
  • org.apache.commons.lang.ArrayUtils => org.apache.commons.lang3.ArrayUtils
  • org.apache.commons.lang.RandomStringUtils => org.apache.commons.lang3.RandomStringUtils
  • org.apache.commons.lang.StringEscapeUtils => org.apache.commons.lang3.StringEscapeUtils
  • org.apache.commons.lang.StringUtils => org.apache.commons.lang3.StringUtils
  • org.apache.commons.lang.SystemUtils => org.apache.commons.lang3.SystemUtils
  • org.apache.commons.lang.exception.ExceptionUtils => org.apache.commons.lang3.exception.ExceptionUtils
  • org.apache.commons.lang.mutable.MutableInt => org.apache.commons.lang3.mutable.MutableInt
  • org.apache.commons.lang.NotImplementedException => org.apache.commons.lang3.NotImplementedException

Aggarwal-Raghav avatar Nov 12 '25 10:11 Aggarwal-Raghav

commons-lang3-dependency-tree.txt Screenshot 2025-11-12 at 4 34 02 PM

Aggarwal-Raghav avatar Nov 12 '25 11:11 Aggarwal-Raghav

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 28m 13s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: xmllint 0m 0s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 12 new or modified test files.
_ master Compile Tests _
+0 :ok: mvndep 2m 32s Maven dependency ordering for branch
+1 :green_heart: mvninstall 10m 0s master passed
+1 :green_heart: compile 7m 9s master passed
+1 :green_heart: checkstyle 4m 27s master passed
+1 :green_heart: javadoc 5m 6s master passed
+0 :ok: spotbugs 2m 14s tez-api in master has 610 extant spotbugs warnings.
+0 :ok: spotbugs 0m 55s tez-runtime-internals in master has 114 extant spotbugs warnings.
+0 :ok: spotbugs 1m 13s tez-runtime-library in master has 241 extant spotbugs warnings.
+0 :ok: spotbugs 0m 57s tez-mapreduce in master has 129 extant spotbugs warnings.
+0 :ok: spotbugs 1m 48s tez-dag in master has 785 extant spotbugs warnings.
+0 :ok: spotbugs 0m 52s tez-tests in master has 6 extant spotbugs warnings.
+0 :ok: spotbugs 0m 48s tez-plugins/tez-history-parser in master has 21 extant spotbugs warnings.
+0 :ok: spotbugs 0m 39s tez-tools/tez-javadoc-tools in master has 3 extant spotbugs warnings.
+0 :ok: spotbugs 7m 31s root in master has 2066 extant spotbugs warnings.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 14s Maven dependency ordering for patch
-1 :x: mvninstall 0m 21s /patch-mvninstall-tez-tests.txt tez-tests in the patch failed.
-1 :x: mvninstall 0m 11s /patch-mvninstall-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in the patch failed.
+1 :green_heart: codespell 0m 58s No new issues.
+1 :green_heart: compile 7m 8s the patch passed
+1 :green_heart: javac 7m 8s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 4m 9s the patch passed
+1 :green_heart: javadoc 4m 58s the patch passed
+1 :green_heart: spotbugs 19m 6s the patch passed
_ Other Tests _
+1 :green_heart: unit 2m 28s tez-api in the patch passed.
+1 :green_heart: unit 1m 5s tez-runtime-internals in the patch passed.
+1 :green_heart: unit 6m 2s tez-runtime-library in the patch passed.
+1 :green_heart: unit 1m 35s tez-mapreduce in the patch passed.
+1 :green_heart: unit 5m 13s tez-dag in the patch passed.
-1 :x: unit 5m 47s /patch-unit-tez-tests.txt tez-tests in the patch passed.
-1 :x: unit 0m 50s /patch-unit-tez-plugins_tez-history-parser.txt tez-history-parser in the patch passed.
+1 :green_heart: unit 0m 28s tez-javadoc-tools in the patch passed.
-1 :x: unit 26m 20s /patch-unit-root.txt root in the patch passed.
+1 :green_heart: asflicense 3m 25s The patch does not generate ASF License warnings.
177m 46s
Reason Tests
Failed junit tests tez.test.TestExceptionPropagation
tez.test.TestAM
tez.test.TestAMRecoveryAggregationBroadcast
tez.test.TestSecureShuffle
tez.mapreduce.TestMRRJobsDAGApi
tez.mapreduce.TestMRRJobs
tez.test.TestTezJobs
tez.test.TestDAGRecovery
tez.test.TestAMRecovery
tez.test.TestFaultTolerance
tez.test.TestPipelinedShuffle
tez.test.TestDAGRecovery2
tez.test.TestRecovery
tez.history.TestHistoryParser
tez.dag.history.ats.acls.TestATSHistoryV15
tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
tez.history.TestHistoryParser
tez.dag.history.ats.acls.TestATSHistoryWithACLs
tez.auxservices.TestShuffleHandlerJobs
tez.analyzer.TestAnalyzer
tez.tests.TestExternalTezServices
tez.tests.TestExternalTezServicesErrors
tez.test.TestExceptionPropagation
tez.test.TestAM
tez.test.TestAMRecoveryAggregationBroadcast
tez.test.TestSecureShuffle
tez.mapreduce.TestMRRJobsDAGApi
tez.mapreduce.TestMRRJobs
tez.test.TestTezJobs
tez.test.TestDAGRecovery
tez.test.TestAMRecovery
tez.test.TestFaultTolerance
tez.test.TestPipelinedShuffle
tez.test.TestDAGRecovery2
tez.test.TestRecovery
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/1/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/441
Optional Tests dupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle
uname Linux d82e17d7ee7d 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/src/.yetus/personality.sh
git revision master / 25ee1ed47cb8abdbfdcba0d131064c200f5f1457
Default Java Ubuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/1/testReport/
Max. process+thread count 2098 (vs. ulimit of 5500)
modules C: tez-api tez-runtime-internals tez-runtime-library tez-mapreduce tez-dag tez-tests tez-plugins/tez-history-parser tez-tools/tez-javadoc-tools . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/1/console
versions git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Nov 12 '25 13:11 tez-yetus

Looks like UT failures are because of exclusion of common-lang 2.x in hadoop-yarn-server-tests jar. I have removed the exclusions. As the jar is in test scope dependency, we are only shipping commons-lang3.x.

From Hive perspective, commons-lang 2.x is part of hive-exec jar so, it shouldn't cause any runtime issues as well.

https://github.com/apache/hive/blob/c054a47c7c22875c93ca0d29b7ac730cf7deba04/ql/pom.xml#L1060

CC @abstractdog

Aggarwal-Raghav avatar Nov 12 '25 16:11 Aggarwal-Raghav

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: xmllint 0m 0s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 12 new or modified test files.
_ master Compile Tests _
+0 :ok: mvndep 2m 54s Maven dependency ordering for branch
+1 :green_heart: mvninstall 10m 48s master passed
+1 :green_heart: compile 7m 7s master passed
+1 :green_heart: checkstyle 4m 31s master passed
+1 :green_heart: javadoc 5m 8s master passed
+0 :ok: spotbugs 2m 15s tez-api in master has 610 extant spotbugs warnings.
+0 :ok: spotbugs 0m 53s tez-runtime-internals in master has 114 extant spotbugs warnings.
+0 :ok: spotbugs 1m 15s tez-runtime-library in master has 241 extant spotbugs warnings.
+0 :ok: spotbugs 0m 57s tez-mapreduce in master has 129 extant spotbugs warnings.
+0 :ok: spotbugs 1m 49s tez-dag in master has 785 extant spotbugs warnings.
+0 :ok: spotbugs 0m 53s tez-tests in master has 6 extant spotbugs warnings.
+0 :ok: spotbugs 0m 48s tez-plugins/tez-history-parser in master has 21 extant spotbugs warnings.
+0 :ok: spotbugs 0m 38s tez-tools/tez-javadoc-tools in master has 3 extant spotbugs warnings.
+0 :ok: spotbugs 7m 31s root in master has 2066 extant spotbugs warnings.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 15s Maven dependency ordering for patch
-1 :x: mvninstall 0m 21s /patch-mvninstall-tez-tests.txt tez-tests in the patch failed.
-1 :x: mvninstall 0m 10s /patch-mvninstall-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in the patch failed.
+1 :green_heart: codespell 1m 0s No new issues.
+1 :green_heart: compile 7m 4s the patch passed
+1 :green_heart: javac 7m 4s the patch passed
+1 :green_heart: blanks 0m 1s The patch has no blanks issues.
+1 :green_heart: checkstyle 4m 9s the patch passed
+1 :green_heart: javadoc 5m 2s the patch passed
+1 :green_heart: spotbugs 18m 22s the patch passed
_ Other Tests _
+1 :green_heart: unit 2m 28s tez-api in the patch passed.
+1 :green_heart: unit 1m 4s tez-runtime-internals in the patch passed.
+1 :green_heart: unit 6m 0s tez-runtime-library in the patch passed.
+1 :green_heart: unit 1m 34s tez-mapreduce in the patch passed.
+1 :green_heart: unit 5m 16s tez-dag in the patch passed.
+1 :green_heart: unit 35m 40s tez-tests in the patch passed.
+1 :green_heart: unit 2m 29s tez-history-parser in the patch passed.
+1 :green_heart: unit 0m 29s tez-javadoc-tools in the patch passed.
+1 :green_heart: unit 69m 37s root in the patch passed.
+1 :green_heart: asflicense 3m 45s The patch does not generate ASF License warnings.
225m 48s
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/2/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/441
Optional Tests dupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle
uname Linux 01a8e1014b78 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/src/.yetus/personality.sh
git revision master / cafecd52c1a60582f1f1382ecb4cf22bfb4d11e3
Default Java Ubuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/2/testReport/
Max. process+thread count 2097 (vs. ulimit of 5500)
modules C: tez-api tez-runtime-internals tez-runtime-library tez-mapreduce tez-dag tez-tests tez-plugins/tez-history-parser tez-tools/tez-javadoc-tools . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/2/console
versions git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Nov 12 '25 20:11 tez-yetus

@Aggarwal-Raghav : I can see only 1 problem in tez-javadoc-tools while compiling the java 17+ version

abstractdog avatar Dec 02 '25 16:12 abstractdog

@Aggarwal-Raghav : I can see only 1 problem in tez-javadoc-tools while compiling the java 17+ version

@abstractdog , it is passing in local. Attaching the build logs for this module. Do i need to force push again? tez-javadoc-buid.txt

Aggarwal-Raghav avatar Dec 02 '25 16:12 Aggarwal-Raghav

Please suggest, what needs to be done.

Aggarwal-Raghav avatar Dec 02 '25 17:12 Aggarwal-Raghav

Please suggest, what needs to be done.

sorry for the late reply, it has passed for me too locally, can you rebase and push to see it now? and then we'll decide

abstractdog avatar Jan 22 '26 11:01 abstractdog

No Worries. Rebased

Aggarwal-Raghav avatar Jan 22 '26 11:01 Aggarwal-Raghav

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: xmllint 0m 0s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 12 new or modified test files.
_ master Compile Tests _
+0 :ok: mvndep 2m 26s Maven dependency ordering for branch
+1 :green_heart: mvninstall 9m 42s master passed
+1 :green_heart: compile 7m 7s master passed
+1 :green_heart: checkstyle 4m 41s master passed
+1 :green_heart: javadoc 5m 28s master passed
-1 :x: spotbugs 1m 17s /branch-spotbugs-tez-api.txt tez-api in master failed.
-1 :x: spotbugs 0m 32s /branch-spotbugs-tez-runtime-internals.txt tez-runtime-internals in master failed.
-1 :x: spotbugs 0m 37s /branch-spotbugs-tez-runtime-library.txt tez-runtime-library in master failed.
-1 :x: spotbugs 0m 36s /branch-spotbugs-tez-mapreduce.txt tez-mapreduce in master failed.
-1 :x: spotbugs 0m 44s /branch-spotbugs-tez-dag.txt tez-dag in master failed.
-1 :x: spotbugs 0m 33s /branch-spotbugs-tez-tests.txt tez-tests in master failed.
-1 :x: spotbugs 0m 32s /branch-spotbugs-tez-plugins_tez-history-parser.txt tez-history-parser in master failed.
-1 :x: spotbugs 0m 26s /branch-spotbugs-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in master failed.
-1 :x: spotbugs 0m 21s /branch-spotbugs-root.txt root in master failed.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 15s Maven dependency ordering for patch
-1 :x: mvninstall 0m 20s /patch-mvninstall-tez-tests.txt tez-tests in the patch failed.
-1 :x: mvninstall 0m 11s /patch-mvninstall-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in the patch failed.
+1 :green_heart: codespell 1m 36s No new issues.
+1 :green_heart: compile 7m 8s the patch passed
+1 :green_heart: javac 7m 8s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 4m 12s the patch passed
+1 :green_heart: javadoc 5m 20s the patch passed
-1 :x: spotbugs 0m 38s /patch-spotbugs-tez-api.txt tez-api in the patch failed.
-1 :x: spotbugs 0m 33s /patch-spotbugs-tez-runtime-internals.txt tez-runtime-internals in the patch failed.
-1 :x: spotbugs 0m 38s /patch-spotbugs-tez-runtime-library.txt tez-runtime-library in the patch failed.
-1 :x: spotbugs 0m 34s /patch-spotbugs-tez-mapreduce.txt tez-mapreduce in the patch failed.
-1 :x: spotbugs 0m 45s /patch-spotbugs-tez-dag.txt tez-dag in the patch failed.
-1 :x: spotbugs 0m 35s /patch-spotbugs-tez-tests.txt tez-tests in the patch failed.
-1 :x: spotbugs 0m 35s /patch-spotbugs-tez-plugins_tez-history-parser.txt tez-history-parser in the patch failed.
-1 :x: spotbugs 0m 29s /patch-spotbugs-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in the patch failed.
-1 :x: spotbugs 0m 24s /patch-spotbugs-root.txt root in the patch failed.
_ Other Tests _
+1 :green_heart: unit 2m 36s tez-api in the patch passed.
+1 :green_heart: unit 1m 4s tez-runtime-internals in the patch passed.
+1 :green_heart: unit 6m 11s tez-runtime-library in the patch passed.
+1 :green_heart: unit 1m 36s tez-mapreduce in the patch passed.
+1 :green_heart: unit 6m 13s tez-dag in the patch passed.
+1 :green_heart: unit 41m 25s tez-tests in the patch passed.
+1 :green_heart: unit 2m 34s tez-history-parser in the patch passed.
+1 :green_heart: unit 0m 30s tez-javadoc-tools in the patch passed.
+1 :green_heart: unit 76m 28s root in the patch passed.
+1 :green_heart: asflicense 4m 4s The patch does not generate ASF License warnings.
213m 16s
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/3/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/441
Optional Tests dupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle
uname Linux 78db6b01deaa 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/src/.yetus/personality.sh
git revision master / c884d63a420694e76aa9ceb68dd9529254b9f0ec
Default Java Ubuntu-21.0.9+10-Ubuntu-124.04
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/3/testReport/
Max. process+thread count 2110 (vs. ulimit of 5500)
modules C: tez-api tez-runtime-internals tez-runtime-library tez-mapreduce tez-dag tez-tests tez-plugins/tez-history-parser tez-tools/tez-javadoc-tools . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/3/console
versions git=2.43.0 maven=3.8.7 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Jan 22 '26 15:01 tez-yetus

looks good to me, spotbugs should be fixed though in TEZ-4669

abstractdog avatar Jan 22 '26 15:01 abstractdog

looks good to me, spotbugs should be fixed though in TEZ-4669

I'll double-check the javadoc tools build, because it's weird...it compiles for me locally too

abstractdog avatar Jan 22 '26 15:01 abstractdog

I'm speculating and might be wrong but is it possible that the -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/yetus-m2/tez-master-patch-0 is gone by the time this ran?

/patch-mvninstall-tez-tests.txt
/patch-mvninstall-tez-tools_tez-javadoc-tools.txt

WARNING] The POM for org.apache.tez:tez-api:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-runtime-library:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-mapreduce:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-dag:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

Aggarwal-Raghav avatar Jan 22 '26 16:01 Aggarwal-Raghav

I'm speculating and might be wrong but is it possible that the -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/yetus-m2/tez-master-patch-0 is gone by the time this ran?

/patch-mvninstall-tez-tests.txt
/patch-mvninstall-tez-tools_tez-javadoc-tools.txt

WARNING] The POM for org.apache.tez:tez-api:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-runtime-library:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-mapreduce:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-dag:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

As its not able to find the previously build modules

Aggarwal-Raghav avatar Jan 22 '26 16:01 Aggarwal-Raghav

I'm speculating and might be wrong but is it possible that the -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/yetus-m2/tez-master-patch-0 is gone by the time this ran?

/patch-mvninstall-tez-tests.txt
/patch-mvninstall-tez-tools_tez-javadoc-tools.txt

WARNING] The POM for org.apache.tez:tez-api:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-runtime-library:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-mapreduce:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-dag:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

As its not able to find the previously build modules

but right before these modules' build, there is e.g. tez-dag build as shown in https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/3/artifact/out/patch-mvninstall-tez-dag.txt

which ends with:

[INFO] Installing /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/src/tez-dag/target/tez-dag-1.0.0-SNAPSHOT.jar to /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/yetus-m2/tez-master-patch-0/org/apache/tez/tez-dag/1.0.0-SNAPSHOT/tez-dag-1.0.0-SNAPSHOT.jar

I cannot reproduce this locally now, and now it's not only tez-javadoc-tools but also tez-tests

more interestingly, tez-runtime-internals builds successfully, even though it's complaining about tez-api, which it depends on: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/3/artifact/out/patch-mvninstall-tez-runtime-internals.txt

[WARNING] The POM for org.apache.tez:tez-api:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

abstractdog avatar Jan 22 '26 16:01 abstractdog

I just hacked yetus to add -X to maven command and explode, will see :) https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/4/console

abstractdog avatar Jan 22 '26 17:01 abstractdog

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 27s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: xmllint 0m 0s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 12 new or modified test files.
_ master Compile Tests _
+0 :ok: mvndep 2m 30s Maven dependency ordering for branch
+1 :green_heart: mvninstall 9m 42s master passed
+1 :green_heart: compile 7m 11s master passed
+1 :green_heart: checkstyle 4m 41s master passed
+1 :green_heart: javadoc 5m 30s master passed
-1 :x: spotbugs 1m 18s /branch-spotbugs-tez-api.txt tez-api in master failed.
-1 :x: spotbugs 0m 33s /branch-spotbugs-tez-runtime-internals.txt tez-runtime-internals in master failed.
-1 :x: spotbugs 0m 37s /branch-spotbugs-tez-runtime-library.txt tez-runtime-library in master failed.
-1 :x: spotbugs 0m 34s /branch-spotbugs-tez-mapreduce.txt tez-mapreduce in master failed.
-1 :x: spotbugs 0m 45s /branch-spotbugs-tez-dag.txt tez-dag in master failed.
-1 :x: spotbugs 0m 35s /branch-spotbugs-tez-tests.txt tez-tests in master failed.
-1 :x: spotbugs 0m 35s /branch-spotbugs-tez-plugins_tez-history-parser.txt tez-history-parser in master failed.
-1 :x: spotbugs 0m 29s /branch-spotbugs-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in master failed.
-1 :x: spotbugs 0m 24s /branch-spotbugs-root.txt root in master failed.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 16s Maven dependency ordering for patch
-1 :x: mvninstall 0m 22s /patch-mvninstall-tez-tests.txt tez-tests in the patch failed.
-1 :x: mvninstall 0m 11s /patch-mvninstall-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in the patch failed.
+1 :green_heart: codespell 1m 37s No new issues.
+1 :green_heart: compile 7m 16s the patch passed
+1 :green_heart: javac 7m 16s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 4m 22s the patch passed
+1 :green_heart: javadoc 5m 23s the patch passed
-1 :x: spotbugs 0m 38s /patch-spotbugs-tez-api.txt tez-api in the patch failed.
-1 :x: spotbugs 0m 33s /patch-spotbugs-tez-runtime-internals.txt tez-runtime-internals in the patch failed.
-1 :x: spotbugs 0m 38s /patch-spotbugs-tez-runtime-library.txt tez-runtime-library in the patch failed.
-1 :x: spotbugs 0m 36s /patch-spotbugs-tez-mapreduce.txt tez-mapreduce in the patch failed.
-1 :x: spotbugs 0m 44s /patch-spotbugs-tez-dag.txt tez-dag in the patch failed.
-1 :x: spotbugs 0m 36s /patch-spotbugs-tez-tests.txt tez-tests in the patch failed.
-1 :x: spotbugs 0m 33s /patch-spotbugs-tez-plugins_tez-history-parser.txt tez-history-parser in the patch failed.
-1 :x: spotbugs 0m 29s /patch-spotbugs-tez-tools_tez-javadoc-tools.txt tez-javadoc-tools in the patch failed.
-1 :x: spotbugs 0m 25s /patch-spotbugs-root.txt root in the patch failed.
_ Other Tests _
+1 :green_heart: unit 2m 36s tez-api in the patch passed.
+1 :green_heart: unit 1m 4s tez-runtime-internals in the patch passed.
+1 :green_heart: unit 6m 15s tez-runtime-library in the patch passed.
+1 :green_heart: unit 1m 35s tez-mapreduce in the patch passed.
+1 :green_heart: unit 6m 8s tez-dag in the patch passed.
+1 :green_heart: unit 36m 57s tez-tests in the patch passed.
+1 :green_heart: unit 2m 35s tez-history-parser in the patch passed.
+1 :green_heart: unit 0m 30s tez-javadoc-tools in the patch passed.
+1 :green_heart: unit 74m 8s root in the patch passed.
+1 :green_heart: asflicense 4m 6s The patch does not generate ASF License warnings.
207m 19s
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/4/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/441
Optional Tests dupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle
uname Linux ce8e27e96d23 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/src/.yetus/personality.sh
git revision master / c884d63a420694e76aa9ceb68dd9529254b9f0ec
Default Java Ubuntu-21.0.9+10-Ubuntu-124.04
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/4/testReport/
Max. process+thread count 1399 (vs. ulimit of 5500)
modules C: tez-api tez-runtime-internals tez-runtime-library tez-mapreduce tez-dag tez-tests tez-plugins/tez-history-parser tez-tools/tez-javadoc-tools . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-441/4/console
versions git=2.43.0 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Jan 22 '26 20:01 tez-yetus

@Aggarwal-Raghav spotbugs report showing -1 for all modules. It is failed due to below error. Can you check it ?


1) null returned by binding at org.eclipse.sisu.wire.LocatorWiring
 but the 1st parameter of org.codehaus.plexus.velocity.internal.DefaultVelocityComponent.<init>(Unknown Source) is not @Nullable
  at org.eclipse.sisu.wire.LocatorWiring
  while locating org.codehaus.plexus.velocity.VelocityComponentConfigurator
    for the 1st parameter of org.codehaus.plexus.velocity.internal.DefaultVelocityComponent.<init>(Unknown Source)
  at ClassRealm[plugin>com.github.spotbugs:spotbugs-maven-plugin:4.9.3.0, parent: jdk.internal.loader.ClassLoaders$AppClassLoader@639fee48] (via modules: org.eclipse.sisu.wire.WireModule -> org.eclipse.sisu.plexus.PlexusBindingModule)
  while locating org.codehaus.plexus.velocity.internal.DefaultVelocityComponent
  while locating java.lang.Object annotated with *
  at org.eclipse.sisu.wire.LocatorWiring
  while locating org.codehaus.plexus.velocity.VelocityComponent
    for field at org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.velocity(Unknown Source)
  at ClassRealm[plugin>com.github.spotbugs:spotbugs-maven-plugin:4.9.3.0, parent: jdk.internal.loader.ClassLoaders$AppClassLoader@639fee48] (via modules: org.eclipse.sisu.wire.WireModule -> org.eclipse.sisu.plexus.PlexusBindingModule)
  while locating org.apache.maven.doxia.siterenderer.DefaultSiteRenderer
  while locating java.lang.Object annotated with *
  while locating org.codehaus.mojo.spotbugs.SpotBugsMojo

1 error
    at com.google.inject.internal.InternalProvisionException.toProvisionException (InternalProvisionException.java:226)
    at com.google.inject.internal.InjectorImpl$1.get (InjectorImpl.java:1097)
    at com.google.inject.internal.InjectorImpl.getInstance (InjectorImpl.java:1131)
    at org.eclipse.sisu.space.AbstractDeferredClass.get (AbstractDeferredClass.java:48)
    at com.google.inject.internal.ProviderInternalFactory.provision (ProviderInternalFactory.java:85)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision (InternalFactoryToInitializableAdapter.java:57)
    at com.google.inject.internal.ProviderInternalFactory$1.call (ProviderInternalFactory.java:66)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:112)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:127)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision (ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ProviderInternalFactory.circularGet (ProviderInternalFactory.java:61)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get (InternalFactoryToInitializableAdapter.java:47)
    at com.google.inject.internal.InjectorImpl$1.get (InjectorImpl.java:1094)

maheshrajus avatar Jan 23 '26 06:01 maheshrajus

@Aggarwal-Raghav spotbugs report showing -1 for all modules. It is failed due to below error. Can you check it ?


1) null returned by binding at org.eclipse.sisu.wire.LocatorWiring
 but the 1st parameter of org.codehaus.plexus.velocity.internal.DefaultVelocityComponent.<init>(Unknown Source) is not @Nullable
  at org.eclipse.sisu.wire.LocatorWiring
  while locating org.codehaus.plexus.velocity.VelocityComponentConfigurator
    for the 1st parameter of org.codehaus.plexus.velocity.internal.DefaultVelocityComponent.<init>(Unknown Source)
  at ClassRealm[plugin>com.github.spotbugs:spotbugs-maven-plugin:4.9.3.0, parent: jdk.internal.loader.ClassLoaders$AppClassLoader@639fee48] (via modules: org.eclipse.sisu.wire.WireModule -> org.eclipse.sisu.plexus.PlexusBindingModule)
  while locating org.codehaus.plexus.velocity.internal.DefaultVelocityComponent
  while locating java.lang.Object annotated with *
  at org.eclipse.sisu.wire.LocatorWiring
  while locating org.codehaus.plexus.velocity.VelocityComponent
    for field at org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.velocity(Unknown Source)
  at ClassRealm[plugin>com.github.spotbugs:spotbugs-maven-plugin:4.9.3.0, parent: jdk.internal.loader.ClassLoaders$AppClassLoader@639fee48] (via modules: org.eclipse.sisu.wire.WireModule -> org.eclipse.sisu.plexus.PlexusBindingModule)
  while locating org.apache.maven.doxia.siterenderer.DefaultSiteRenderer
  while locating java.lang.Object annotated with *
  while locating org.codehaus.mojo.spotbugs.SpotBugsMojo

1 error
    at com.google.inject.internal.InternalProvisionException.toProvisionException (InternalProvisionException.java:226)
    at com.google.inject.internal.InjectorImpl$1.get (InjectorImpl.java:1097)
    at com.google.inject.internal.InjectorImpl.getInstance (InjectorImpl.java:1131)
    at org.eclipse.sisu.space.AbstractDeferredClass.get (AbstractDeferredClass.java:48)
    at com.google.inject.internal.ProviderInternalFactory.provision (ProviderInternalFactory.java:85)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision (InternalFactoryToInitializableAdapter.java:57)
    at com.google.inject.internal.ProviderInternalFactory$1.call (ProviderInternalFactory.java:66)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:112)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:127)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision (ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ProviderInternalFactory.circularGet (ProviderInternalFactory.java:61)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get (InternalFactoryToInitializableAdapter.java:47)
    at com.google.inject.internal.InjectorImpl$1.get (InjectorImpl.java:1094)

already tracked on TEZ-4669

abstractdog avatar Jan 23 '26 08:01 abstractdog

debug level maven logs revealed new exotic things:

[WARNING] Invalid POM for org.apache.tez:tez-dag:jar:1.0.0-SNAPSHOT, transitive dependencies (if any) will not be available, enable debug logging for more details: Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for org.apache.commons:commons-lang3:jar is missing. @ line 106, column 17

so invalid POM is some missing version, but still don't get how could this come in precommit and not locally

abstractdog avatar Jan 23 '26 09:01 abstractdog

reproduced the maven compile problem on an ubuntu server by hitting the same commands as yetus:

mvn clean

cd tez-api && mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true && cd ..
cd tez-dag && mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true && cd ..
cd tez-mapreduce && mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true && cd ..
cd tez-runtime-library && mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true && cd ..
cd tez-runtime-internals && mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true && cd ..
cd tez-tests && mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true && cd ..

again, the same didn't happen on my mac

abstractdog avatar Jan 23 '26 09:01 abstractdog

I'll also look into this over the weekend.

Aggarwal-Raghav avatar Jan 23 '26 09:01 Aggarwal-Raghav

Can we remove this cd and replace with -pland try once? I'm building that docker image in build-tools/docker but its taking time.

mvn clean install -pl tez-api,tez-dag,tez-mapreduce,tez-runtime-library,tez-runtime-internals,tez-tests \
    -DskipTests \
    -Dmaven.javadoc.skip=true \
    -Dcheckstyle.skip=true \
    -Dfindbugs.skip=true \
    -Dspotbugs.skip=true

Aggarwal-Raghav avatar Jan 23 '26 10:01 Aggarwal-Raghav

Can we remove this cd and replace with -pland try once? I'm building that docker image in build-tools/docker but its taking time.

mvn clean install -pl tez-api,tez-dag,tez-mapreduce,tez-runtime-library,tez-runtime-internals,tez-tests \
    -DskipTests \
    -Dmaven.javadoc.skip=true \
    -Dcheckstyle.skip=true \
    -Dfindbugs.skip=true \
    -Dspotbugs.skip=true

I'm afraid we cannot, this is deep inside yetus logic, how it identifies the affected modules and their dependencies, and build them one by one

abstractdog avatar Jan 23 '26 10:01 abstractdog

ignore my last comment about effective pom, it was wrong, deleted it

abstractdog avatar Jan 23 '26 13:01 abstractdog

hm, strange, now I can repro the same locally too :D

mvn -version

Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/lbodor/Applications/maven
Java version: 21.0.8, vendor: Eclipse Adoptium, runtime: /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home
Default locale: en_HU, platform encoding: UTF-8
OS name: "mac os x", version: "15.7.3", arch: "aarch64", family: "mac"


mvn clean
rm -rf ~/.m2/repository/org/apache/tez

cd tez-api && mvn clean install -DskipTests && cd ..
cd tez-runtime-internals && mvn clean install -DskipTests && cd ..
cd tez-runtime-library && mvn clean install -DskipTests && cd ..
cd tez-mapreduce && mvn clean install -DskipTests && cd ..
cd tez-dag && mvn clean install -DskipTests && cd ..
cd tez-tests && mvn clean install -DskipTests && cd ..

abstractdog avatar Jan 23 '26 14:01 abstractdog

I'm also able to repro with the steps you have shared on mac but @abstractdog , isn't this way of doing things wrong? if clean the ~/.m2/repository cache and build the modules like tez-runtime-internals the depedent modules (tez-commons )etc. will be downloaded from snapshot artifactory. We shouldn't delete the .m2 for this smart-apply-patch.sh

Aggarwal-Raghav avatar Jan 23 '26 14:01 Aggarwal-Raghav

I'm speculating and might be wrong but is it possible that the -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/yetus-m2/tez-master-patch-0 is gone by the time this ran?

/patch-mvninstall-tez-tests.txt
/patch-mvninstall-tez-tools_tez-javadoc-tools.txt

WARNING] The POM for org.apache.tez:tez-api:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-runtime-library:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-mapreduce:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-dag:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

As its not able to find the previously build modules

That's what I was suspecting yesterday as well. Maybe I couldn't communicate it better.

Aggarwal-Raghav avatar Jan 23 '26 14:01 Aggarwal-Raghav

I'm speculating and might be wrong but is it possible that the -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-441/yetus-m2/tez-master-patch-0 is gone by the time this ran?

/patch-mvninstall-tez-tests.txt
/patch-mvninstall-tez-tools_tez-javadoc-tools.txt

WARNING] The POM for org.apache.tez:tez-api:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-runtime-library:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-mapreduce:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.tez:tez-dag:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

As its not able to find the previously build modules

That's what I was suspecting yesterday as well. Maybe I couldn't communicate it better.

I agree that building the project module by module is not necessarily the most optimal way, because we can figure out what went wrong even if the whole project is built in one go, I'm totally fine with a patch where the maven build phase is done in one go, and as a positive side effect, this issue would also go away

abstractdog avatar Jan 23 '26 16:01 abstractdog

made a decision here, this PR should be merged, I'm not about to block valuable contributions because we cannot figure out and solve infra problems easily, so in this case:

  1. spotbugs is broken: TEZ-4669
  2. mvn install is broken if there is a new property in the root pom: TEZ-4684

this patch looks safe and clean

abstractdog avatar Jan 26 '26 09:01 abstractdog