hadoop
hadoop copied to clipboard
apache
HADOOP-18508. Support multiple s3a integration test runs on same bucket in parallel
- adds job id to fork id
- adds option to disable root tests
- documents this
- gets more tests working under the job/fork subdir rather than absolute paths. tracking them down is tricky.
How was this patch tested?
running the forked tests with different job ids, disabling root dir cleanup and seeing what was created in separate subdirs
For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
s3 london
time mvit -Dparallel-tests -DtestsThreadCount=10 -Dscale -Dfs.s3a.root.tests.enabled=false -Djob.id=001
where mvit is mvn verify with some options
need to run a few more times to make sure all files are created in isolation. then kick off two test runs in different source trees at the same time
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 54s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 13 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 38s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 52s | trunk passed | |
| +1 :green_heart: | compile | 24m 58s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 21m 36s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 4m 1s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 27s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 35s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 18s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 3m 52s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 39s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 32s | the patch passed | |
| +1 :green_heart: | compile | 24m 21s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 24m 21s | the patch passed | |
| +1 :green_heart: | compile | 21m 29s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 21m 29s | the patch passed | |
| -1 :x: | blanks | 0m 0s | /blanks-eol.txt | The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 48s | /results-checkstyle-root.txt | root: The patch generated 1 new + 86 unchanged - 0 fixed = 87 total (was 86) |
| +1 :green_heart: | mvnsite | 2m 26s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 18s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 4m 5s | the patch passed | |
| +1 :green_heart: | shadedclient | 24m 10s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 18m 59s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 39s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. | |
| 239m 37s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 089d080102eb 4.15.0-192-generic #203-Ubuntu SMP Wed Aug 10 17:40:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 409f306c50ae95a734610c9377b53f048d72ef2b |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/1/testReport/ |
| Max. process+thread count | 1239 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
rebased; running as an experiment.
the jobid stuff is all set up in the pom.xml. i actually want to be able to set it in the auth-keys.xml file, as that would let me set up different source trees testing against different parts of the same store.
proposed: support it as a sysprop or a config option also, review all uses of createTestPath() to make sure they do the right thing.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 48s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 13 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 16m 48s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 48s | trunk passed | |
| +1 :green_heart: | compile | 19m 40s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 17m 39s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 52s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 42s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 56s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 38s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 52s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 21s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 27s | the patch passed | |
| +1 :green_heart: | compile | 18m 49s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 49s | the patch passed | |
| +1 :green_heart: | compile | 17m 45s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 45s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 43s | root: The patch generated 0 new + 85 unchanged - 2 fixed = 85 total (was 87) | |
| +1 :green_heart: | mvnsite | 2m 34s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 47s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 40s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 4m 12s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 33s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 30s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 3m 1s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 270m 22s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 067e10f15349 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aaf0c32c58b08c42cc642343f50ba3c9f871b511 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/7/testReport/ |
| Max. process+thread count | 1329 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Can I chase @mukund-thakur @HarshitGupta11 @ahmarsuhail for reviews here. This is so we can run multiple CI tests against the same bucket in parallel.
Are you running multiple parallel tests in different terminal windows to verify everything is correct?
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 17m 52s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 46s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 32m 32s | trunk passed | |
| +1 :green_heart: | compile | 17m 28s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 16m 7s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 26s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 38s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 57s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 47s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 55s | trunk passed | |
| +1 :green_heart: | shadedclient | 34m 33s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 27s | the patch passed | |
| +1 :green_heart: | compile | 16m 47s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 16m 47s | the patch passed | |
| +1 :green_heart: | compile | 16m 8s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 16m 8s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 22s | /results-checkstyle-root.txt | root: The patch generated 8 new + 151 unchanged - 2 fixed = 159 total (was 153) |
| +1 :green_heart: | mvnsite | 2m 42s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 53s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 45s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 4m 16s | the patch passed | |
| +1 :green_heart: | shadedclient | 34m 57s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 20m 47s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 3m 19s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 265m 28s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 4983acf2ab6b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 63a7ed0822d061d5632a7d7138a7d31e63149d28 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/8/testReport/ |
| Max. process+thread count | 3151 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
...and I've found out that terasort paths aren't isolated
[ERROR] Failures:
[ERROR] org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A.test_120_terasort[magic](org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A)
[ERROR] Run 1: ITestTerasortOnS3A.test_120_terasort:289->executeStage:239->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 terasort(s3a://stevel-london/terasort-magic-false/sortin, s3a://stevel-london/terasort-magic-false/sortout) failed expected:<0> but was:<1>
[ERROR] Run 2: ITestTerasortOnS3A.test_120_terasort:289->executeStage:239->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 terasort(s3a://stevel-london/terasort-magic-true/sortin, s3a://stevel-london/terasort-magic-true/sortout) failed expected:<0> but was:<1>
[INFO]
[ERROR] ITestTerasortOnS3A.test_130_teravalidate:305->executeStage:239->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teravalidate(s3a://stevel-london/terasort-directory-false/sortout, s3a://stevel-london/terasort-directory-false/validate) failed expected:<0> but was:<1>
[INFO]
[ERROR] Tests run: 141, Failures: 2, Errors: 0, Skipped: 46
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 18m 20s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 1s | The patch appears to include 17 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 20s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 38m 57s | trunk passed | |
| +1 :green_heart: | compile | 19m 39s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 17m 45s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 50s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 39s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 54s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 54s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 56s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 29s | the patch passed | |
| +1 :green_heart: | compile | 18m 51s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 51s | the patch passed | |
| +1 :green_heart: | compile | 17m 35s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 35s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 54s | /results-checkstyle-root.txt | root: The patch generated 8 new + 159 unchanged - 2 fixed = 167 total (was 161) |
| +1 :green_heart: | mvnsite | 2m 36s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 48s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 4m 13s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 57s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 20m 34s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 3m 1s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | |
| 289m 58s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 224e3c033534 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ad85edda7c00e61b40ad231a03c0fea8cda55ed7 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/9/testReport/ |
| Max. process+thread count | 3136 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
checkstyles where I changed the base contract tests
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java:25:import org.junit.Assert;:8: Unused import - org.junit.Assert. [UnusedImports]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java:27:import org.junit.Test;:8: Unused import - org.junit.Test. [UnusedImports]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java:22:import java.io.IOException;:8: Unused import - java.io.IOException. [UnusedImports]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java:27:import org.apache.hadoop.fs.Path;:8: Unused import - org.apache.hadoop.fs.Path. [UnusedImports]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java:29:import org.junit.Assert;:8: Unused import - org.junit.Assert. [UnusedImports]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java:31:import org.junit.Test;:8: Unused import - org.junit.Test. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFSMainOperations.java:24:import org.junit.Assert;:8: Unused import - org.junit.Assert. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFSMainOperations.java:26:import org.junit.Test;:8: Unused import - org.junit.Test. [UnusedImports]
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 04s | No case conflicting files found. | |
| +0 :ok: | spotbugs | 0m 00s | spotbugs executables are not available. | |
| +0 :ok: | codespell | 0m 00s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 00s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 00s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 00s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 01s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 00s | The patch appears to include 17 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 2m 19s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 90m 47s | trunk passed | |
| +1 :green_heart: | compile | 41m 33s | trunk passed | |
| +1 :green_heart: | checkstyle | 6m 17s | trunk passed | |
| +1 :green_heart: | mvnsite | 12m 55s | trunk passed | |
| +1 :green_heart: | javadoc | 9m 55s | trunk passed | |
| +1 :green_heart: | shadedclient | 167m 23s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 2m 21s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 8m 08s | the patch passed | |
| +1 :green_heart: | compile | 38m 28s | the patch passed | |
| +1 :green_heart: | javac | 38m 28s | the patch passed | |
| +1 :green_heart: | blanks | 0m 00s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 6m 08s | the patch passed | |
| +1 :green_heart: | mvnsite | 12m 47s | the patch passed | |
| +1 :green_heart: | javadoc | 10m 55s | the patch passed | |
| +1 :green_heart: | shadedclient | 182m 40s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 6m 04s | The patch does not generate ASF License warnings. | |
| 546m 41s |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | MINGW64_NT-10.0-17763 7a8d877045a1 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / ad85edda7c00e61b40ad231a03c0fea8cda55ed7 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-5081/3/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-5081/3/console |
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 49s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 17 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 17s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 28s | trunk passed | |
| +1 :green_heart: | compile | 19m 14s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 17m 36s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 48s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 34s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 52s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 54s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 31s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 27s | the patch passed | |
| +1 :green_heart: | compile | 18m 27s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 27s | the patch passed | |
| +1 :green_heart: | compile | 17m 43s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 43s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 49s | /results-checkstyle-root.txt | root: The patch generated 4 new + 159 unchanged - 2 fixed = 163 total (was 161) |
| +1 :green_heart: | mvnsite | 2m 31s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 47s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 4m 14s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 35s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 20m 30s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 58s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 268m 44s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 8f94e011b5ba 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d2247a411373988606dd7780df6b40ff8f13181b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/10/testReport/ |
| Max. process+thread count | 2583 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 48s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 17 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 0s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 49s | trunk passed | |
| +1 :green_heart: | compile | 19m 13s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 17m 48s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 50s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 40s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 52s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 38s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 52s | trunk passed | |
| +1 :green_heart: | shadedclient | 39m 40s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed | |
| +1 :green_heart: | compile | 18m 38s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javac | 18m 38s | the patch passed | |
| +1 :green_heart: | compile | 17m 40s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 17m 40s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 48s | /results-checkstyle-root.txt | root: The patch generated 4 new + 159 unchanged - 2 fixed = 163 total (was 161) |
| +1 :green_heart: | mvnsite | 2m 36s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 47s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | javadoc | 1m 40s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 4m 13s | the patch passed | |
| +1 :green_heart: | shadedclient | 39m 52s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 20m 23s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 55s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 269m 28s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux d2e3ec8702ce 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 23e696c5de713a6c627a6fa190942c11f8b5c2cb |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/11/testReport/ |
| Max. process+thread count | 3136 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5081/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@mukund-thakur retested, identified a race condition in terasorts and fixed. all should be good now
I checked out this branch yesterday and ran tests in parallel in two terminals and saw many failures. Could be my setup issue as well but let's wait on this till I figure out what is going on.
This is continuously failing. `[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC [ERROR] Tests run: 24, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 110.261 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC [ERROR] testListEncryptedDirkeep-markers Time elapsed: 1.399 s <<< ERROR! java.io.FileNotFoundException: No such file or directory: s3a://bucket/job-001/test/testListEncryptedDir/a at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4095) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3946) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:5247) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$40(S3AFileSystem.java:5178) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449) at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2716) at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2735) at org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:5177) at org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC.testListEncryptedDir(ITestS3AEncryptionSSEC.java:242) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750)
[ERROR] testListEncryptedDirdelete-markers Time elapsed: 1.67 s <<< ERROR! java.io.FileNotFoundException: No such file or directory: s3a://bucket/job-001/test/testListEncryptedDir/a at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4095) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3946) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:5247) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$40(S3AFileSystem.java:5178) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449) at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2716) at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2735) at org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:5177) at org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC.testListEncryptedDir(ITestS3AEncryptionSSEC.java:242)`
- everything good on a single terminal test run?
- you were using different copies of the hadoop source tree?
- everything good on a single terminal test run?
above error on even a single test run as well. Not related to parallelism.
- you were using different copies of the hadoop source tree?
I just figured out while debugging that we should be on different copies of hadoop source tree. So I think we should mention this in doc.
I have SSE-KMS with key configured in auth-keys.xml
and command mvn -Dparallel-tests -DtestsThreadCount=8 clean verify -Dtest=none -Djob.id=001
Re-ran parallelly in two terminals and I think we are good apart from the above failure. and the above test passes in my trunk branch.
These two tests are failing on trunk as well.
[ERROR] testEverything(org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory) Time elapsed: 6.661 s <<< FAILURE! java.lang.AssertionError: Expected a org.apache.hadoop.fs.s3a.commit.PathCommitException to be thrown, but got the result: : FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202405210976_0008}; taskId=attempt_202405210976_0008_m_000000_0, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter@41f68cab}; outputPath=s3a://mthakur-us-west-1/job-002-fork-0008/test/testEverything, workPath=s3a://mthakur-us-west-1/job-002-fork-0008/test/testEverything/_temporary/1/_temporary/attempt_202405210976_0008_m_000000_0, algorithmVersion=1, skipCleanup=false, ignoreCleanupFailures=false} at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:453) at org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory.testInvalidFileBinding(ITestS3ACommitterFactory.java:165) at org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory.testEverything(ITestS3ACommitterFactory.java:115)
[ERROR] testS3SpecificSignerOverride(org.apache.hadoop.fs.s3a.ITestS3AConfiguration) Time elapsed: 1.692 s <<< ERROR! software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@21daa1bb: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@74d904d3: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@47d37b27: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@297a12fd: Unable to contact EC2 metadata service.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.regions.providers.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:70) at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveRegion(AwsDefaultClientBuilder.java:293) at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:600) at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:589) at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:396) at software.amazon.awssdk.utils.AttributeMap$Builder.internalGet(AttributeMap.java:389) at software.amazon.awssdk.utils.AttributeMap$Builder.access$1300(AttributeMap.java:201) at software.amazon.awssdk.utils.AttributeMap$Builder$1.get(AttributeMap.java:399) at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveEndpoint(AwsDefaultClientBuilder.java:268) at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:600) at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:589) at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:396) at java.util.ArrayList.forEach(ArrayList.java:1259) at software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362) at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:232) at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeAwsConfiguration(AwsDefaultClientBuilder.java:184) at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeChildConfiguration(AwsDefaultClientBuilder.java:161) at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:188) at software.amazon.awssdk.services.sts.DefaultStsClientBuilder.buildClient(DefaultStsClientBuilder.java:36) at software.amazon.awssdk.services.sts.DefaultStsClientBuilder.buildClient(DefaultStsClientBuilder.java:25) at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:155) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testS3SpecificSignerOverride(ITestS3AConfiguration.java:579)
I need to sit down and fix that committer factory one. nothing hard but it is irritating me
..it and the STS one are both with jiras. STS failure depends on the region you are running from, so it is test setup.
@mukund-thakur given the failures you saw were just related to you not reading the docs, can we merge this?
@mukund-thakur given the failures you saw were just related to you not reading the docs, can we merge this? Haha sure apart from this https://github.com/apache/hadoop/pull/5081#issuecomment-2121184602 LGTM +1
let me rebase and look at the failing test before I merge. You are only seeing it on this branch?
Draft message for the commit
It is now possible to run s3a integration tests from different source trees/computers/VMs
against a single shared bucket by
* disabling the tests which modify the entire bucket (-Dfs.s3a.root.tests.enabled=false)
* passing in a job ID property "-Djob.id=001" or setting this in auth-keys.xml
Consult hadoop-aws testing documentation for the details
This is intended for CI builds. Although it also permits parallel test runs in different
local copies of the hadoop source tree, maven will guarantee that your classpath
will contain the latest version of any module compiled, irrespective of which
source tree it came from.
Contributed by Steve Loughran
@mukund-thakur couldn't replicate the failure but I'd disabled the tests during some s3express testing and not-reenabled it. still couldn't replicate it.
and when you skip root tests then it's also disabled as mixing SSE-C really causes problems. Or at least used to.
Anyway, I've made sure the suite sets a base dir at the start of each methodPath() and builds each sub path off it, whatever problem you encountered seems to have been related.
I also cut out the markers=delete run phase as its markers=keep where more problems surface...this saves 30s off the test suite.
Which makes me think: we should review when we use markers=delete and other than for some compatibility tests, remove them all. What do you think about having more time in your life back?