hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard

HDFS-17299. Adding rack failure tolerance when creating a new file

Open ritegarg opened this issue 1 year ago • 8 comments

Description of PR

How was this patch tested?

For code changes:

  • [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

ritegarg avatar Mar 06 '24 23:03 ritegarg

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 46s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 20s branch-3.3 passed
+1 :green_heart: compile 2m 20s branch-3.3 passed
+1 :green_heart: checkstyle 0m 38s branch-3.3 passed
+1 :green_heart: mvnsite 1m 34s branch-3.3 passed
+1 :green_heart: javadoc 1m 36s branch-3.3 passed
-1 :x: spotbugs 1m 28s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 11s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 48s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 24s the patch passed
+1 :green_heart: compile 2m 11s the patch passed
+1 :green_heart: javac 2m 11s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 32s hadoop-hdfs-project: The patch generated 0 new + 249 unchanged - 3 fixed = 249 total (was 252)
+1 :green_heart: mvnsite 1m 21s the patch passed
+1 :green_heart: javadoc 1m 25s the patch passed
+1 :green_heart: spotbugs 3m 16s the patch passed
+1 :green_heart: shadedclient 21m 57s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 46s hadoop-hdfs-client in the patch passed.
-1 :x: unit 172m 4s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 31s The patch does not generate ASF License warnings.
276m 34s
Reason Tests
Failed junit tests hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
hadoop.hdfs.server.datanode.TestLargeBlockReport
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/1/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2bf19f76095a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f4fad11cc92defc7568c0658648dfc04899ff180
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/1/testReport/
Max. process+thread count 4872 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 07 '24 03:03 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 22s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 12m 59s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 27s branch-3.3 passed
+1 :green_heart: compile 2m 14s branch-3.3 passed
+1 :green_heart: checkstyle 0m 37s branch-3.3 passed
+1 :green_heart: mvnsite 1m 27s branch-3.3 passed
+1 :green_heart: javadoc 1m 37s branch-3.3 passed
-1 :x: spotbugs 1m 26s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 23m 12s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 21s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 18s the patch passed
+1 :green_heart: compile 2m 9s the patch passed
+1 :green_heart: javac 2m 9s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 33s hadoop-hdfs-project: The patch generated 0 new + 249 unchanged - 3 fixed = 249 total (was 252)
+1 :green_heart: mvnsite 1m 20s the patch passed
+1 :green_heart: javadoc 1m 18s the patch passed
+1 :green_heart: spotbugs 3m 18s the patch passed
+1 :green_heart: shadedclient 22m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 49s hadoop-hdfs-client in the patch passed.
-1 :x: unit 172m 34s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 31s The patch does not generate ASF License warnings.
276m 57s
Reason Tests
Failed junit tests hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.server.mover.TestMover
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.datanode.TestLargeBlockReport
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/2/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 8029685ad3de 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 5d4a6ed957d86f85618f70f27d11f6077336b16f
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/2/testReport/
Max. process+thread count 4424 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 07 '24 10:03 hadoop-yetus

There are few test failures. Can you please take a look? @ritegarg

shahrs87 avatar Mar 07 '24 17:03 shahrs87

There are few test failures. Can you please take a look? @ritegarg

I was looking into the failures, looks like transient failures. The same tests are running fine locally.

ritegarg avatar Mar 07 '24 19:03 ritegarg

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 1s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 44s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 14s branch-3.3 passed
+1 :green_heart: compile 2m 16s branch-3.3 passed
+1 :green_heart: checkstyle 0m 38s branch-3.3 passed
+1 :green_heart: mvnsite 1m 31s branch-3.3 passed
+1 :green_heart: javadoc 1m 37s branch-3.3 passed
-1 :x: spotbugs 1m 27s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 23s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 22s the patch passed
+1 :green_heart: compile 2m 13s the patch passed
+1 :green_heart: javac 2m 13s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 31s hadoop-hdfs-project: The patch generated 0 new + 249 unchanged - 3 fixed = 249 total (was 252)
+1 :green_heart: mvnsite 1m 18s the patch passed
+1 :green_heart: javadoc 1m 20s the patch passed
+1 :green_heart: spotbugs 3m 26s the patch passed
+1 :green_heart: shadedclient 22m 3s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 47s hadoop-hdfs-client in the patch passed.
-1 :x: unit 121m 34s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+0 :ok: asflicense 0m 27s ASF License check generated no output?
225m 17s
Reason Tests
Failed junit tests hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
hadoop.hdfs.TestEncryptedTransfer
hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
hadoop.hdfs.server.datanode.TestBatchIbr
hadoop.hdfs.server.datanode.TestBlockScanner
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy
hadoop.hdfs.server.datanode.TestDataNodeFaultInjector
hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
hadoop.hdfs.server.datanode.TestBlockRecovery2
hadoop.hdfs.TestParallelUnixDomainRead
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/3/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux e6f838e8b04d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 2cfb16244810628ba6c4c6b1282a3f50568c302a
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/3/testReport/
Max. process+thread count 4738 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/3/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 07 '24 20:03 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 3m 51s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 14m 43s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 41s branch-3.3 passed
+1 :green_heart: compile 2m 15s branch-3.3 passed
+1 :green_heart: checkstyle 0m 39s branch-3.3 passed
+1 :green_heart: mvnsite 1m 29s branch-3.3 passed
+1 :green_heart: javadoc 1m 37s branch-3.3 passed
-1 :x: spotbugs 1m 29s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 34s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 23s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 22s the patch passed
+1 :green_heart: compile 2m 13s the patch passed
+1 :green_heart: javac 2m 13s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 29s hadoop-hdfs-project: The patch generated 0 new + 249 unchanged - 3 fixed = 249 total (was 252)
+1 :green_heart: mvnsite 1m 20s the patch passed
+1 :green_heart: javadoc 1m 23s the patch passed
+1 :green_heart: spotbugs 3m 16s the patch passed
+1 :green_heart: shadedclient 22m 1s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 46s hadoop-hdfs-client in the patch passed.
-1 :x: unit 174m 18s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 31s The patch does not generate ASF License warnings.
283m 20s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/4/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux b7a1a36bac6b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 39ce2d70cc78ba50870927ca7d2235fa25e00c66
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/4/testReport/
Max. process+thread count 4387 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/4/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 09 '24 22:03 hadoop-yetus

Ran the tests locally and observing that some of the failures are not reproducible and tests are passing. 1 test is failing with and without changes. 1 test is flaky in both upstream and PR.

Test Name With Changes(locally) Without Changes
TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart Pass Pass
TestBalancerWithHANameNodes.testBalancerWithObserverWithFailedNode Flaky Flaky
TestReconstructStripedBlocksWithRackAwareness.testChooseExcessReplicasToDelete Pass Pass
TestDataNodeRollingUpgrade.testDatanodeRollingUpgradeWithFinalize Pass Pass
TestDirectoryScanner.testThrottling Pass Pass
TestLargeBlockReport.testBlockReportSucceedsWithLargerLengthLimit Fail Fail
TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart Pass Pass

ritegarg avatar Mar 09 '24 23:03 ritegarg

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 3m 46s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 18s Maven dependency ordering for branch
+1 :green_heart: mvninstall 23m 0s branch-3.3 passed
+1 :green_heart: compile 2m 19s branch-3.3 passed
+1 :green_heart: checkstyle 0m 38s branch-3.3 passed
+1 :green_heart: mvnsite 1m 35s branch-3.3 passed
+1 :green_heart: javadoc 1m 36s branch-3.3 passed
-1 :x: spotbugs 1m 31s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 20s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 47s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 23s the patch passed
+1 :green_heart: compile 2m 10s the patch passed
+1 :green_heart: javac 2m 10s the patch passed
-1 :x: blanks 0m 0s /blanks-eol.txt The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 :green_heart: checkstyle 0m 34s hadoop-hdfs-project: The patch generated 0 new + 249 unchanged - 3 fixed = 249 total (was 252)
+1 :green_heart: mvnsite 1m 21s the patch passed
+1 :green_heart: javadoc 1m 24s the patch passed
+1 :green_heart: spotbugs 3m 21s the patch passed
+1 :green_heart: shadedclient 22m 14s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 46s hadoop-hdfs-client in the patch passed.
-1 :x: unit 172m 33s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 32s The patch does not generate ASF License warnings.
281m 22s
Reason Tests
Failed junit tests hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.server.datanode.TestLargeBlockReport
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/5/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 8885b87773d0 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / fa769f6cfa7845ef46da6a862c6ccab699956446
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/5/testReport/
Max. process+thread count 4406 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/5/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 10 '24 03:03 hadoop-yetus

@ritegarg Thank you for the PR. Overall looks good. Looks like blanks and spotbugs warnings are relevant. Can you please fix them? Once that is done, I will approve and merge.

shahrs87 avatar Mar 13 '24 16:03 shahrs87

@ritegarg Thank you for the PR. Overall looks good. Looks like blanks and spotbugs warnings are relevant. Can you please fix them? Once that is done, I will approve and merge.

I see 2 spotbugs failures

  1. Possible null dereference in DFSOutputStream.java:[line 314] -> There is a PreConditions.checkNotNull before that in line 312
  2. Redundant Null Check of possible null values PeerCache.java:[line 158] -> Removed the null check

Also removed blank line

ritegarg avatar Mar 13 '24 16:03 ritegarg

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 44s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 42s branch-3.3 passed
+1 :green_heart: compile 2m 17s branch-3.3 passed
+1 :green_heart: checkstyle 0m 38s branch-3.3 passed
+1 :green_heart: mvnsite 1m 29s branch-3.3 passed
+1 :green_heart: javadoc 1m 40s branch-3.3 passed
-1 :x: spotbugs 1m 28s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 22s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 21s the patch passed
+1 :green_heart: compile 2m 12s the patch passed
+1 :green_heart: javac 2m 12s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 28s hadoop-hdfs-project: The patch generated 0 new + 254 unchanged - 3 fixed = 254 total (was 257)
+1 :green_heart: mvnsite 1m 20s the patch passed
+1 :green_heart: javadoc 1m 25s the patch passed
+1 :green_heart: spotbugs 1m 28s hadoop-hdfs-project/hadoop-hdfs-client generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2)
+1 :green_heart: spotbugs 1m 53s hadoop-hdfs in the patch passed.
+1 :green_heart: shadedclient 22m 7s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 46s hadoop-hdfs-client in the patch passed.
-1 :x: unit 173m 2s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 32s The patch does not generate ASF License warnings.
277m 51s
Reason Tests
Failed junit tests hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.server.datanode.TestLargeBlockReport
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/6/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux ddde9c9242be 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 0bf2223760f96a5b370b2caf7e92fa95ba8a72de
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/6/testReport/
Max. process+thread count 4392 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/6/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 13 '24 21:03 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 1s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 26s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 37s branch-3.3 passed
+1 :green_heart: compile 2m 19s branch-3.3 passed
+1 :green_heart: checkstyle 0m 40s branch-3.3 passed
+1 :green_heart: mvnsite 1m 27s branch-3.3 passed
+1 :green_heart: javadoc 1m 34s branch-3.3 passed
-1 :x: spotbugs 1m 31s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 22s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 21s the patch passed
+1 :green_heart: compile 2m 13s the patch passed
+1 :green_heart: javac 2m 13s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 32s hadoop-hdfs-project: The patch generated 0 new + 254 unchanged - 3 fixed = 254 total (was 257)
+1 :green_heart: mvnsite 1m 20s the patch passed
+1 :green_heart: javadoc 1m 22s the patch passed
+1 :green_heart: spotbugs 3m 21s the patch passed
+1 :green_heart: shadedclient 22m 11s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 45s hadoop-hdfs-client in the patch passed.
-1 :x: unit 172m 13s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 30s The patch does not generate ASF License warnings.
276m 37s
Reason Tests
Failed junit tests hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.server.datanode.TestLargeBlockReport
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/7/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 6b8506aa5106 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 44afa1412ebfda61c31d82d9c492e4aa044b6089
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/7/testReport/
Max. process+thread count 4496 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/7/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 14 '24 02:03 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 3m 49s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 7s Maven dependency ordering for branch
+1 :green_heart: mvninstall 24m 22s branch-3.3 passed
+1 :green_heart: compile 2m 21s branch-3.3 passed
+1 :green_heart: checkstyle 0m 42s branch-3.3 passed
+1 :green_heart: mvnsite 1m 34s branch-3.3 passed
+1 :green_heart: javadoc 1m 40s branch-3.3 passed
-1 :x: spotbugs 1m 35s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 25s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 23s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 28s the patch passed
+1 :green_heart: compile 2m 17s the patch passed
+1 :green_heart: javac 2m 17s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 34s hadoop-hdfs-project: The patch generated 0 new + 254 unchanged - 3 fixed = 254 total (was 257)
+1 :green_heart: mvnsite 1m 17s the patch passed
+1 :green_heart: javadoc 1m 23s the patch passed
+1 :green_heart: spotbugs 3m 48s the patch passed
+1 :green_heart: shadedclient 22m 25s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 50s hadoop-hdfs-client in the patch passed.
-1 :x: unit 177m 13s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 28s The patch does not generate ASF License warnings.
287m 38s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
hadoop.hdfs.TestReadStripedFileWithDecoding
hadoop.hdfs.TestReconstructStripedFileWithValidator
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
hadoop.hdfs.TestViewDistributedFileSystem
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.tools.TestDFSAdmin
hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
hadoop.hdfs.server.balancer.TestBalancer
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/8/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 87f15e31dfb0 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f780ddf7235805441de364bbc4e9385b2e414527
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/8/testReport/
Max. process+thread count 4123 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/8/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 14 '24 02:03 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 2m 45s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 6s Maven dependency ordering for branch
+1 :green_heart: mvninstall 23m 28s branch-3.3 passed
+1 :green_heart: compile 2m 21s branch-3.3 passed
+1 :green_heart: checkstyle 0m 39s branch-3.3 passed
+1 :green_heart: mvnsite 1m 30s branch-3.3 passed
+1 :green_heart: javadoc 1m 42s branch-3.3 passed
-1 :x: spotbugs 1m 36s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 15s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 22s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 21s the patch passed
+1 :green_heart: compile 2m 30s the patch passed
+1 :green_heart: javac 2m 30s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 30s hadoop-hdfs-project: The patch generated 0 new + 254 unchanged - 3 fixed = 254 total (was 257)
+1 :green_heart: mvnsite 1m 19s the patch passed
+1 :green_heart: javadoc 1m 24s the patch passed
+1 :green_heart: spotbugs 3m 42s the patch passed
+1 :green_heart: shadedclient 22m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 46s hadoop-hdfs-client in the patch passed.
-1 :x: unit 183m 10s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 31s The patch does not generate ASF License warnings.
291m 33s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.TestDFSStripedInputStream
hadoop.hdfs.TestFileChecksumCompositeCrc
hadoop.hdfs.tools.TestDFSAdmin
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.tools.TestECAdmin
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/9/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 0d63f43cdbae 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f780ddf7235805441de364bbc4e9385b2e414527
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/9/testReport/
Max. process+thread count 4136 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/9/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 14 '24 02:03 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 :ok: mvndep 13m 21s Maven dependency ordering for branch
+1 :green_heart: mvninstall 22m 40s branch-3.3 passed
+1 :green_heart: compile 2m 18s branch-3.3 passed
+1 :green_heart: checkstyle 0m 37s branch-3.3 passed
+1 :green_heart: mvnsite 1m 32s branch-3.3 passed
+1 :green_heart: javadoc 1m 35s branch-3.3 passed
-1 :x: spotbugs 1m 28s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart: shadedclient 22m 18s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 22s Maven dependency ordering for patch
+1 :green_heart: mvninstall 1m 20s the patch passed
+1 :green_heart: compile 2m 12s the patch passed
+1 :green_heart: javac 2m 12s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 29s hadoop-hdfs-project: The patch generated 0 new + 254 unchanged - 3 fixed = 254 total (was 257)
+1 :green_heart: mvnsite 1m 20s the patch passed
+1 :green_heart: javadoc 1m 18s the patch passed
+1 :green_heart: spotbugs 3m 16s the patch passed
+1 :green_heart: shadedclient 22m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 48s hadoop-hdfs-client in the patch passed.
-1 :x: unit 174m 2s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 31s The patch does not generate ASF License warnings.
277m 34s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.TestBlocksScheduledCounter
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.server.namenode.TestNameNodeMXBean
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/11/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6612
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 89ea6b6b9885 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f780ddf7235805441de364bbc4e9385b2e414527
Default Java Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/11/testReport/
Max. process+thread count 4500 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6612/11/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 14 '24 21:03 hadoop-yetus

All the failed test in this build are flaky. Merging this PR. Thank you @ritegarg for your contribution !

shahrs87 avatar Mar 14 '24 22:03 shahrs87