hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard

MAPREDUCE-7402. fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.

Open KeeProMise opened this issue 11 months ago • 13 comments

Description of PR

see also: https://issues.apache.org/jira/browse/MAPREDUCE-7402 In the method :
long computeBytesInMerges(int factor, int inMem), when factor is set to 1, and the initial numSegments>1, inMem=0, it will cause an infinite loop of while. Although setting factor to 1 is meaningless, and it will not be done in actual production, we need to remind users to avoid an infinite loop due to carelessness.

image

How was this patch tested?

new UT testInMemoryAndOnDiskMerger()

For code changes:

  • [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

KeeProMise avatar Mar 11 '24 15:03 KeeProMise

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 32m 44s trunk passed
+1 :green_heart: compile 0m 25s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 20s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 26s trunk passed
+1 :green_heart: mvnsite 0m 28s trunk passed
+1 :green_heart: javadoc 0m 26s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 19s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 52s trunk passed
+1 :green_heart: shadedclient 19m 27s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 21s the patch passed
+1 :green_heart: compile 0m 18s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 18s the patch passed
+1 :green_heart: compile 0m 19s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 19s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 17s the patch passed
+1 :green_heart: mvnsite 0m 20s the patch passed
+1 :green_heart: javadoc 0m 12s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 14s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 50s the patch passed
+1 :green_heart: shadedclient 19m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 5m 29s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 24s The patch does not generate ASF License warnings.
86m 12s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/1/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux b7f308976a60 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 540460637ad3e1dd115eb967eef11d4d6a9b321f
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/1/testReport/
Max. process+thread count 1125 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 11 '24 16:03 hadoop-yetus

@ayushtkn @tomicooler Hi, do you have time to help review this PR? thanks~

KeeProMise avatar Mar 12 '24 02:03 KeeProMise

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 32m 15s trunk passed
+1 :green_heart: compile 0m 25s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 23s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 25s trunk passed
+1 :green_heart: mvnsite 0m 29s trunk passed
+1 :green_heart: javadoc 0m 24s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 20s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 52s trunk passed
+1 :green_heart: shadedclient 19m 43s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 20s the patch passed
+1 :green_heart: compile 0m 19s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 19s the patch passed
+1 :green_heart: compile 0m 16s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 16s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 16s /results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 89 unchanged - 0 fixed = 90 total (was 89)
+1 :green_heart: mvnsite 0m 19s the patch passed
+1 :green_heart: javadoc 0m 14s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 15s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 53s the patch passed
+1 :green_heart: shadedclient 19m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 5m 34s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 25s The patch does not generate ASF License warnings.
86m 31s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/2/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux e87213e1045c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ea7635ffaa001610007801913de7af90bc37cff3
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/2/testReport/
Max. process+thread count 1560 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 12 '24 12:03 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 32m 8s trunk passed
+1 :green_heart: compile 0m 26s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 23s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 24s trunk passed
+1 :green_heart: mvnsite 0m 27s trunk passed
+1 :green_heart: javadoc 0m 26s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 18s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 53s trunk passed
+1 :green_heart: shadedclient 19m 42s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 21s the patch passed
+1 :green_heart: compile 0m 20s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 20s the patch passed
+1 :green_heart: compile 0m 19s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 19s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 17s the patch passed
+1 :green_heart: mvnsite 0m 20s the patch passed
+1 :green_heart: javadoc 0m 14s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 14s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 50s the patch passed
+1 :green_heart: shadedclient 21m 26s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 5m 44s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 24s The patch does not generate ASF License warnings.
88m 27s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/3/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux f353da2d86a4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 30c888e79489d73ffe367235561e0d89f9b29cce
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/3/testReport/
Max. process+thread count 1645 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 12 '24 14:03 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 32m 30s trunk passed
+1 :green_heart: compile 0m 26s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 22s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 26s trunk passed
+1 :green_heart: mvnsite 0m 26s trunk passed
+1 :green_heart: javadoc 0m 25s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 21s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 55s trunk passed
+1 :green_heart: shadedclient 19m 25s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 21s the patch passed
+1 :green_heart: compile 0m 19s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 19s the patch passed
+1 :green_heart: compile 0m 18s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 18s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 17s the patch passed
+1 :green_heart: mvnsite 0m 20s the patch passed
+1 :green_heart: javadoc 0m 14s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 15s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 0m 53s the patch passed
+1 :green_heart: shadedclient 19m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 5m 30s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 25s The patch does not generate ASF License warnings.
85m 57s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/4/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux eb0103e9975c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8c892cf16df7c3411204676617ca05f5015da65e
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/4/testReport/
Max. process+thread count 1609 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 13 '24 13:03 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 33s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 43m 50s trunk passed
+1 :green_heart: compile 0m 41s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 41s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 43s trunk passed
+1 :green_heart: mvnsite 0m 46s trunk passed
+1 :green_heart: javadoc 0m 35s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 28s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 26s trunk passed
+1 :green_heart: shadedclient 33m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 32s the patch passed
+1 :green_heart: compile 0m 32s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 32s the patch passed
+1 :green_heart: compile 0m 31s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 31s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 30s the patch passed
+1 :green_heart: mvnsite 0m 34s the patch passed
+1 :green_heart: javadoc 0m 20s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 20s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 25s the patch passed
+1 :green_heart: shadedclient 33m 7s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 7m 41s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 39s The patch does not generate ASF License warnings.
132m 48s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/5/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 965852b5ffa9 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8c892cf16df7c3411204676617ca05f5015da65e
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/5/testReport/
Max. process+thread count 1632 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 13 '24 14:03 hadoop-yetus

Was gonna hit the merge button, but got curious.

Earlier if the value was 1, was it an infinite loop always? or in just in a corner case, if in just one corner case & now if we throw an exception, if anyone has value of 1 earlier his jobs would be working but post this change it would fail?

can we just break the condition for infinite loop if that is the case & preserve the value? or at worst if that is an illegal value, we put a warn log & use the defaul?

@brumi1024 / @szilard-nemeth any pointers?

Hi. The code related to this method has been running for a very long time (since June 13, 2011), and I suspect that setting this value to 1 is meaningless in the actual generation environment. Therefore, this bug should have existed for a long time, but has not been reported yet. Therefore, I believe that if the value is 1, throwing an exception will not have any impact on the existing business.

KeeProMise avatar Mar 30 '24 03:03 KeeProMise

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
_ Prechecks _
+1 :green_heart: dupname 0m 00s No case conflicting files found.
+0 :ok: spotbugs 0m 01s spotbugs executables are not available.
+0 :ok: codespell 0m 01s codespell was not available.
+0 :ok: detsecrets 0m 01s detect-secrets was not available.
+1 :green_heart: @author 0m 01s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 00s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 128m 20s trunk passed
+1 :green_heart: compile 7m 45s trunk passed
+1 :green_heart: checkstyle 7m 29s trunk passed
+1 :green_heart: mvnsite 7m 56s trunk passed
+1 :green_heart: javadoc 7m 14s trunk passed
+1 :green_heart: shadedclient 206m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 4m 20s the patch passed
+1 :green_heart: compile 3m 52s the patch passed
+1 :green_heart: javac 3m 52s the patch passed
+1 :green_heart: blanks 0m 00s The patch has no blanks issues.
+1 :green_heart: checkstyle 3m 15s the patch passed
+1 :green_heart: mvnsite 3m 55s the patch passed
+1 :green_heart: javadoc 3m 13s the patch passed
+1 :green_heart: shadedclient 218m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: asflicense 8m 21s The patch does not generate ASF License warnings.
588m 58s
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname MINGW64_NT-10.0-17763 44a06cd45f54 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool maven
Personality /c/hadoop/dev-support/bin/hadoop.sh
git revision trunk / 8c892cf16df7c3411204676617ca05f5015da65e
Default Java Azul Systems, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/1/testReport/
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/1/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 25 '24 01:04 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 9m 12s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 44m 30s trunk passed
+1 :green_heart: compile 0m 38s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 36s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: checkstyle 0m 42s trunk passed
+1 :green_heart: mvnsite 0m 43s trunk passed
+1 :green_heart: javadoc 0m 33s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 27s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 29s trunk passed
+1 :green_heart: shadedclient 34m 36s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 32s the patch passed
+1 :green_heart: compile 0m 33s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 33s the patch passed
+1 :green_heart: compile 0m 31s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: javac 0m 31s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 31s the patch passed
+1 :green_heart: mvnsite 0m 36s the patch passed
+1 :green_heart: javadoc 0m 20s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 20s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 34s the patch passed
+1 :green_heart: shadedclient 34m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 7m 27s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 36s The patch does not generate ASF License warnings.
144m 52s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/7/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 8ccb3c9abd48 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / dff82fdd3ca73a2fd2b8b9c33330a97f50135f26
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/7/testReport/
Max. process+thread count 1639 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/7/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar May 08 '24 05:05 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 13m 6s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 44m 6s trunk passed
+1 :green_heart: compile 0m 42s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 35s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: checkstyle 0m 43s trunk passed
+1 :green_heart: mvnsite 0m 47s trunk passed
+1 :green_heart: javadoc 0m 34s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 28s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 33s trunk passed
+1 :green_heart: shadedclient 34m 47s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 33s the patch passed
+1 :green_heart: compile 0m 35s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 35s the patch passed
+1 :green_heart: compile 0m 32s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: javac 0m 32s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 32s the patch passed
+1 :green_heart: mvnsite 0m 36s the patch passed
+1 :green_heart: javadoc 0m 19s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 20s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 36s the patch passed
+1 :green_heart: shadedclient 34m 32s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 7m 27s hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 0m 36s The patch does not generate ASF License warnings.
148m 42s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/6/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux e57bc58d7d20 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / dff82fdd3ca73a2fd2b8b9c33330a97f50135f26
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/6/testReport/
Max. process+thread count 1568 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6622/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar May 08 '24 05:05 hadoop-yetus

Was gonna hit the merge button, but got curious.

Earlier if the value was 1, was it an infinite loop always? or in just in a corner case, if in just one corner case & now if we throw an exception, if anyone has value of 1 earlier his jobs would be working but post this change it would fail?

can we just break the condition for infinite loop if that is the case & preserve the value? or at worst if that is an illegal value, we put a warn log & use the defaul?

@brumi1024 / @szilard-nemeth any pointers? @ayushtkn hi, to prevent throwing exceptions from affecting the task, I will print a warning log instead of throwing exceptions. Please take a look again, thanks.

KeeProMise avatar May 08 '24 15:05 KeeProMise

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
_ Prechecks _
+1 :green_heart: dupname 0m 00s No case conflicting files found.
+0 :ok: spotbugs 0m 01s spotbugs executables are not available.
+0 :ok: codespell 0m 01s codespell was not available.
+0 :ok: detsecrets 0m 01s detect-secrets was not available.
+1 :green_heart: @author 0m 00s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 00s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 108m 33s trunk passed
+1 :green_heart: compile 5m 55s trunk passed
+1 :green_heart: checkstyle 5m 29s trunk passed
+1 :green_heart: mvnsite 6m 14s trunk passed
+1 :green_heart: javadoc 5m 25s trunk passed
+1 :green_heart: shadedclient 175m 55s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 32s the patch passed
+1 :green_heart: compile 2m 49s the patch passed
+1 :green_heart: javac 2m 49s the patch passed
+1 :green_heart: blanks 0m 01s The patch has no blanks issues.
+1 :green_heart: checkstyle 2m 29s the patch passed
+1 :green_heart: mvnsite 3m 09s the patch passed
+1 :green_heart: javadoc 2m 27s the patch passed
+1 :green_heart: shadedclient 187m 58s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 :x: unit 14m 29s /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 9m 11s The patch does not generate ASF License warnings.
526m 12s
Reason Tests
Failed junit tests hadoop.mapred.TestTaskProgressReporter
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname MINGW64_NT-10.0-17763 cf5d9f159693 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool maven
Personality /c/hadoop/dev-support/bin/hadoop.sh
git revision trunk / dff82fdd3ca73a2fd2b8b9c33330a97f50135f26
Default Java Azul Systems, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/3/testReport/
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/3/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar May 10 '24 05:05 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
_ Prechecks _
+1 :green_heart: dupname 0m 00s No case conflicting files found.
+0 :ok: spotbugs 0m 01s spotbugs executables are not available.
+0 :ok: codespell 0m 01s codespell was not available.
+0 :ok: detsecrets 0m 01s detect-secrets was not available.
+1 :green_heart: @author 0m 00s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 00s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 92m 25s trunk passed
+1 :green_heart: compile 5m 10s trunk passed
+1 :green_heart: checkstyle 5m 00s trunk passed
+1 :green_heart: mvnsite 5m 14s trunk passed
+1 :green_heart: javadoc 4m 41s trunk passed
+1 :green_heart: shadedclient 147m 57s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 49s the patch passed
+1 :green_heart: compile 2m 35s the patch passed
+1 :green_heart: javac 2m 35s the patch passed
+1 :green_heart: blanks 0m 00s The patch has no blanks issues.
+1 :green_heart: checkstyle 2m 05s the patch passed
+1 :green_heart: mvnsite 2m 34s the patch passed
+1 :green_heart: javadoc 2m 05s the patch passed
+1 :green_heart: shadedclient 158m 58s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 :x: unit 12m 00s /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core in the patch passed.
+1 :green_heart: asflicense 5m 39s The patch does not generate ASF License warnings.
442m 08s
Reason Tests
Failed junit tests hadoop.mapred.TestTaskProgressReporter
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hadoop/pull/6622
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname MINGW64_NT-10.0-17763 759ee4f0c44d 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool maven
Personality /c/hadoop/dev-support/bin/hadoop.sh
git revision trunk / dff82fdd3ca73a2fd2b8b9c33330a97f50135f26
Default Java Azul Systems, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/4/testReport/
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/4/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar May 10 '24 05:05 hadoop-yetus