hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-19093. [ABFS] ILoadTestRenameRecovery
Description of PR
HADOOP-19093. Adds new test which can be explicitly launched to try and trigger rename throttling.
How was this patch tested?
It doesn't generate enough load at least when executed outside of azure infrastructure.
I'm not sure the best way to go here. I think ideally I'd actually like an azure storage a/c with significantly lower capacity than the default 10K IOPS so as validate resilience better
For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 8s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 31m 40s | trunk passed | |
| +1 :green_heart: | compile | 17m 54s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 16m 19s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 4m 21s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 39s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 5s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 43s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| -1 :x: | spotbugs | 2m 36s | /branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html | hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 33m 50s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 27s | the patch passed | |
| +1 :green_heart: | compile | 16m 40s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 16m 40s | the patch passed | |
| +1 :green_heart: | compile | 16m 22s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 16m 22s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 22s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 36s | the patch passed | |
| +1 :green_heart: | javadoc | 2m 0s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 44s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 4m 11s | the patch passed | |
| +1 :green_heart: | shadedclient | 33m 54s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 38s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 40s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 244m 18s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5523a5e5e5ee 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e250fffa39e6eb1d637882b0b6c0d8e6fdf6a238 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/1/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
When mapreduce.manifest.committer.validate.output I am going to add pre-rename validation phase we scan the task attempt to directory tree to make sure everything is there and with the expected etag. This is simply a repetition of the tree walk.
I'm going to add rate limiting for all IO within the committer. Currently there's a rate limiter for the committer rename -in the FS instance, so shared across all threads- but nothing else.
my plan is in hadoop common to add a new interface RateLimiterSource where you can then ask for the getRateLimiter(String Path, String operation); with well-known operation names. ABFS Will provide this API and just return the existing rename limiter everywhere. In S3A I'll add separate limiters for read and write operations which can be configured for the 5000 and 3500 limits under a prefix. The WiP bulk delete API #6494 will use this for its rate limiting.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | docker | 5m 6s | Docker failed to build run-specific yetus/hadoop:tp-4955}. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/2/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 48s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 41s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 48s | trunk passed | |
| +1 :green_heart: | compile | 19m 30s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 18m 36s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 53s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 35s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 59s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 19s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 25s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 10s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 39m 38s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 53s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 59s | the patch passed | |
| +1 :green_heart: | compile | 18m 43s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 43s | the patch passed | |
| +1 :green_heart: | compile | 18m 6s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 18m 6s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 43s | /results-checkstyle-root.txt | root: The patch generated 2 new + 23 unchanged - 0 fixed = 25 total (was 23) |
| -1 :x: | mvnsite | 0m 58s | /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| +1 :green_heart: | javadoc | 2m 43s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 20s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 55s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 20s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 30s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 7m 47s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 2m 49s | hadoop-azure in the patch passed. | |
| -1 :x: | asflicense | 1m 2s | /results-asflicense.txt | The patch generated 1 ASF License warnings. |
| 285m 43s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 540c94ea5d03 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 662723f8bf333192731e3a910ee26006d4cb57b6 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/3/testReport/ |
| Max. process+thread count | 2542 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
testing on an overloaded azure store; hit sime timeouts and an assertion failure in org.apache.hadoop.fs.azurebfs.ITestAbfsClient.testListPathWithValueGreaterThanServerMaximum(ITestAbfsClient.java; created a new JIRA there as it looks like an assertion tuning only.
Latest PR adds op-specific capacity multiplication to abfs store; numbers are pretty arbitrary as this is only used in the committer. could be tuned in future.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 49s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 31s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 11s | trunk passed | |
| +1 :green_heart: | compile | 19m 47s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 18m 6s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 54s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 34s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 45s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 19s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 25s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 33s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 40m 0s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 59s | the patch passed | |
| +1 :green_heart: | compile | 18m 37s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 37s | the patch passed | |
| +1 :green_heart: | compile | 17m 49s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 49s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 41s | /results-checkstyle-root.txt | root: The patch generated 3 new + 23 unchanged - 0 fixed = 26 total (was 23) |
| -1 :x: | mvnsite | 0m 56s | /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| +1 :green_heart: | javadoc | 2m 40s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 19s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 56s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 3s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 42s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 7m 40s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 2m 47s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. | |
| 284m 15s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux baf8f559fa8e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b3092b69ca21a43d00a9d8a6f5f8451fe9a696f1 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/4/testReport/ |
| Max. process+thread count | 1264 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
I am having fun testing at scale with a throttled store.
[ERROR] testSeekZeroByteFile(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek) Time elapsed: 0.339 s <<< ERROR!
java.io.IOException: Failed with java.io.IOException while processing file/directory :[/fork-0003/test/seekfile.txt] in method:[Operation failed: "The resource was created or modified by the Azure Blob Service API and cannot be written to by the Azure Data Lake Storage Service API.", 409, PUT, https://stevelukwest.dfs.core.windows.net/stevel-testing/fork-0003/test/seekfile.txt?action=flush&retainUncommittedData=false&position=1024&close=true&timeout=90, rId: 2c806902-b01f-0087-20c1-7fec44000000, InvalidFlushOperation, "The resource was created or modified by the Azure Blob Service API and cannot be written to by the Azure Data Lake Storage Service API. RequestId:2c806902-b01f-0087-20c1-7fec44000000 Time:2024-03-26T21:04:12.1799296Z"]
- likely regression from the manifest changes
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.484 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[ERROR] testPrepareDirtyTree(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage) Time elapsed: 9.767 s <<< FAILURE!
java.lang.AssertionError: Expected a java.io.IOException to be thrown, but got the result: : Result{directory count=64}
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.TestCreateOutputDirectoriesStage.testPrepareDirtyTree(TestCreateOutputDirectoriesStage.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
[ERROR] test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time elapsed: 27.232 s <<< ERROR!
java.io.IOException:
java.io.IOException: Unknown Job job_1711487009902_0002
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:240)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:254)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:287)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1249)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1172)
will need to isolate
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 48s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 14s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 15s | trunk passed | |
| +1 :green_heart: | compile | 19m 34s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 18m 45s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 47s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 35s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 48s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 19s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 25s | trunk passed | |
| +1 :green_heart: | shadedclient | 38m 47s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 39m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 58s | the patch passed | |
| +1 :green_heart: | compile | 18m 47s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 47s | the patch passed | |
| +1 :green_heart: | compile | 17m 56s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 56s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 6m 14s | /results-checkstyle-root.txt | root: The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) |
| -1 :x: | mvnsite | 0m 56s | /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvnsite | 0m 54s | /patch-mvnsite-hadoop-tools_hadoop-azure.txt | hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 42s | /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt | hadoop-azure in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 46s | /results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 generated 4 new + 15 unchanged - 0 fixed = 19 total (was 15) |
| -1 :x: | spotbugs | 0m 48s | /patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 42s | /patch-spotbugs-hadoop-tools_hadoop-azure.txt | hadoop-azure in the patch failed. |
| +1 :green_heart: | shadedclient | 40m 27s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 35s | hadoop-common in the patch passed. | |
| -1 :x: | unit | 0m 48s | /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | unit | 0m 42s | /patch-unit-hadoop-tools_hadoop-azure.txt | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 275m 37s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux acf764745178 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 70fe8c4aa42d6813cf5e87020d2a5f26e2aeb36c |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/5/testReport/ |
| Max. process+thread count | 1264 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 49s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 43s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 13s | trunk passed | |
| +1 :green_heart: | compile | 19m 38s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 18m 2s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 50s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 36s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 48s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 20s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 22s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 14s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 39m 42s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed | |
| +1 :green_heart: | compile | 18m 43s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 43s | the patch passed | |
| +1 :green_heart: | compile | 17m 50s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 50s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 44s | /results-checkstyle-root.txt | root: The patch generated 4 new + 23 unchanged - 0 fixed = 27 total (was 23) |
| -1 :x: | mvnsite | 0m 57s | /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| +1 :green_heart: | javadoc | 2m 41s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 20s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 54s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 8s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 39s | hadoop-common in the patch passed. | |
| -1 :x: | unit | 7m 45s | /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 2m 48s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 284m 58s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.mapreduce.lib.output.committer.manifest.TestCreateOutputDirectoriesStage |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux ee70095a1f64 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4eb685208926b8fc1d1bb2b8047823bfd9652efc |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/6/testReport/ |
| Max. process+thread count | 1855 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 52s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 32m 20s | trunk passed | |
| +1 :green_heart: | compile | 17m 38s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 16m 14s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 27s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 39s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 47s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 27s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 5m 29s | trunk passed | |
| +1 :green_heart: | shadedclient | 34m 2s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 34m 29s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 3s | the patch passed | |
| +1 :green_heart: | compile | 16m 48s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 16m 48s | the patch passed | |
| +1 :green_heart: | compile | 16m 16s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 16m 16s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 18s | /results-checkstyle-root.txt | root: The patch generated 3 new + 23 unchanged - 0 fixed = 26 total (was 23) |
| -1 :x: | mvnsite | 0m 58s | /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch failed. |
| +1 :green_heart: | javadoc | 2m 41s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 2m 27s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 6m 1s | the patch passed | |
| +1 :green_heart: | shadedclient | 34m 44s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 50s | hadoop-common in the patch passed. | |
| -1 :x: | unit | 7m 37s | /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 262m 44s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.mapreduce.lib.output.committer.manifest.TestCreateOutputDirectoriesStage |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 65cc0a53e790 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 80462c51a49ed5a407760a012f2aa9e7fca659b7 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/7/testReport/ |
| Max. process+thread count | 2152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6596/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.