hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-19258: Upgrade aws sdk v2 to 2.27.14
Description of PR
Upgrading aws sdk to 2.27.14
How was this patch tested?
Tested with maven in us-east-1
For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [x] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 17m 58s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 9s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 5s | trunk passed | |
| +1 :green_heart: | compile | 19m 39s | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | compile | 18m 10s | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | mvnsite | 26m 36s | trunk passed | |
| +1 :green_heart: | javadoc | 9m 13s | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javadoc | 8m 7s | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | shadedclient | 55m 57s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 2m 19s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 34m 50s | the patch passed | |
| +1 :green_heart: | compile | 19m 34s | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javac | 19m 34s | the patch passed | |
| +1 :green_heart: | compile | 18m 13s | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | javac | 18m 13s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | mvnsite | 20m 29s | the patch passed | |
| +1 :green_heart: | shellcheck | 0m 0s | No new issues. | |
| +1 :green_heart: | javadoc | 9m 38s | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javadoc | 7m 52s | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | shadedclient | 57m 18s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 820m 6s | /patch-unit-root.txt | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 39s | The patch does not generate ASF License warnings. | |
| 1168m 11s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain | |
| hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage | |
| hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown | |
| hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities | |
| hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps | |
| hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown | |
| hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun | |
| hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction | |
| hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7015 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux f3cb20736a6e 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / efa0fc65e993e15b1a9f4b5e14794df8fdddcbac |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/1/testReport/ |
| Max. process+thread count | 3680 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
- what command line options did you pass to the test run. e.g (-Dscale, ...)
- did you do a test with s3 express or with any third party store
- session credential/role tests (set fs.s3a.assumed.role.arn to a valid ARN For this)
- did you do the full testing.md stuff about command line manual testing, and looking through the logs for new error messages
- did any new dependencies sneak out even though bundle.jar is meant to be shaded?
we have to be really rigorous here as there are many regressions which can sneak past the test suites (new log messages on abort(), performance slow downs) which aren't picked up by junit itself. And needless exported dependencies is a recurrent PITA.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 30s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 32m 54s | trunk passed | |
| +1 :green_heart: | compile | 17m 50s | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | compile | 16m 18s | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | checkstyle | 4m 23s | trunk passed | |
| +1 :green_heart: | mvnsite | 25m 55s | trunk passed | |
| +1 :green_heart: | javadoc | 9m 30s | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javadoc | 8m 9s | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +0 :ok: | spotbugs | 0m 21s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| -1 :x: | spotbugs | 31m 40s | /branch-spotbugs-root-warnings.html | root in trunk has 2 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 66m 10s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 1m 51s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 30m 56s | the patch passed | |
| +1 :green_heart: | compile | 17m 13s | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javac | 17m 13s | the patch passed | |
| +1 :green_heart: | compile | 16m 1s | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | javac | 16m 2s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 23s | the patch passed | |
| +1 :green_heart: | mvnsite | 19m 27s | the patch passed | |
| +1 :green_heart: | shellcheck | 0m 0s | No new issues. | |
| +1 :green_heart: | javadoc | 8m 45s | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javadoc | 8m 9s | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +0 :ok: | spotbugs | 0m 20s | hadoop-project has no data from spotbugs | |
| +1 :green_heart: | shadedclient | 64m 33s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 757m 25s | root in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 42s | The patch does not generate ASF License warnings. | |
| 1136m 50s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7015 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 666f9c344838 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d8530d316e44a4dc4dd1c084ec7eb77fe8a90318 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/2/testReport/ |
| Max. process+thread count | 3964 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Testing against a Regular S3 bucket in us-east-1:
auth-keys
<configuration>
<property>
<name>test.fs.s3a.name</name>
<value>s3a://<REDACTED>/</value>
</property>
<property>
<name>fs.contract.test.fs.s3a</name>
<value>s3a://<REDACTED>/</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<description>AWS access key ID. Omit for IAM role-based authentication.</description>
<value><REDACTED></value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>AWS secret key. Omit for IAM role-based authentication.</description>
<value><REDACTED></value>
</property>
<property>
<name>fs.s3a.encryption.key</name>
<value>arn:aws:kms:us-east-1:<REDACTED>:key/<REDACTED></value>
</property>
<property>
<name>fs.s3a.assumed.role.arn</name>
<value>arn:aws:iam::<REDACTED>:role/<REDACTED></value>
</property>
<property>
<name>test.fs.s3a.sts.enabled</name>
<value>false</value>
</property>
</configuration>
Scale Tests
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 3, Time elapsed: 8.89 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration) Time elapsed: 0.029 s <<< ERROR!
java.lang.IllegalArgumentException: Cannot locate field clientConfiguration on class software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient
at org.apache.commons.lang3.Validate.isTrue(Validate.java:158)
at org.apache.commons.lang3.reflect.FieldUtils.readField(FieldUtils.java:469)
at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.getField(ITestS3AConfiguration.java:535)
at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:449)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 169.585 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[ERROR] testThreadPoolCoolDown(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps) Time elapsed: 143.941 s <<< FAILURE!
java.lang.AssertionError: Failed to find threads in active FS - test is flawed. Actual: 0
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failEquals(Assert.java:187)
at org.junit.Assert.assertNotEquals(Assert.java:201)
at org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.testThreadPoolCoolDown(ITestS3AConcurrentOps.java:206)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR] Failures:
[ERROR] ITestS3AConcurrentOps.testThreadPoolCoolDown:206->Assert.assertNotEquals:201->Assert.failEquals:187->Assert.fail:89 Failed to find threads in active FS - test is flawed. Actual: 0
[ERROR] Errors:
[ERROR] ITestS3AConfiguration.testRequestTimeout:449->getField:535 » IllegalArgument C...
[INFO]
[ERROR] Tests run: 1327, Failures: 1, Errors: 1, Skipped: 127
adding access point:
<property>
<name>fs.s3a.bucket.sample-bucket.accesspoint.arn</name>
<value> <REDACTED> </value>
<description>Configure S3a traffic to use this AccessPoint</description>
</property>
mvn clean verify -Dit.test=ITest* -Dtest=none
Getting a lot of errors, currently debugging:
several parsing errors
grep "Failed to parse b-00" itest_log| sort | uniq -c
132 java.lang.Exception: Failed to parse b-00
[^[[1;31mERROR^[[m] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics) Time elapsed: 2.179 s <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but was:<512>
[^[[1;31mERROR^[[m] testClosedInstrumentation(org.apache.hadoop.fs.s3a.ITestS3AClosedFS) Time elapsed: 0.333 s <<< FAILURE!
org.junit.ComparisonFailure: [S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
few encryption errors:
[^[[1;31mERROR^[[m] testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 2.502 s <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected expected:<kms+context> but was:<null>
[^[[1;31mERROR^[[m] testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 3.498 s <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
<[0, 1, 239, 4079]>
to contain exactly in any order:
<[0, 1, 255, 4095]>
elements not found:
<[255, 4095]>
and elements not expected:
<[239, 4079]>
[^[[1;31mERROR^[[m] testBytesReadWithStream(org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic) Time elapsed: 1.646 s <<< FAILURE!
java.lang.AssertionError: Mismatch in number of FS bytes read by InputStreams expected:<2048> but was:<196167>
[^[[1;31mERROR^[[m] testCustomSignerAndInitializer[simple-delete](org.apache.hadoop.fs.s3a.auth.ITestCustomSigner) Time elapsed: 1.138 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://s3atestingdsgrewal/job-00/test/testCustomSignerAndInitializer[simple-delete]/customsignerpath1: software.amazon.awssdk.services.s3.model.S3Exception: The authorization mechanism you have provided is not supported. Please use Signature Version 4. (Service: S3, Status Code: 400, Request ID: Z68EWFD946XZRPW9, Extended Request ID: 4oF/6V21FWJZdR5osr6DcNYC6RLxEQZrNxEf3/X8ur0Tm/aeVe31aNtZXT/VU/yrG17ZMUOAZK4=):InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4. (Service: S3, Status Code: 400, Request ID: Z68EWFD946XZRPW9, Extended Request ID: 4oF/6V21FWJZdR5osr6DcNYC6RLxEQZrNxEf3/X8ur0Tm/aeVe31aNtZXT/VU/yrG17ZMUOAZK4=)
[^[[1;31mERROR^[[m] testComplexDirActions(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List) Time elapsed: 16.345 s <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>
[^[[1;31mERROR^[[m] testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename) Time elapsed: 2.337 s <<< FAILURE!
java.lang.AssertionError: Number of bytes copied: files_copied_bytes expected:<16384> but was:<16368>
[^[[1;31mERROR^[[m] ^[[1;31m ITestS3AClosedFS.testClosedInstrumentation:111 [S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>^[[m
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4235] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
ILoadTest
mvn verify -Dtest=skip -Dit.test=ILoadTest* -Dscale > loadtest.txt
[INFO] Running org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 200.663 s - in org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.692 s - in org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.984 s - in org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials
regarding the others
- ITestS3AConcurrentOps. hit that in #6938 where I'd assumed it was my change that did it: maybe it just guaranteed it would also surface.
- ITestS3AConfiguration looks like its due to a private field change. If you can see what it was changed to we could migrate the test case -if not, it'll need cutting. Other ones I'm unsure about -could be race conditions based on where you are, or its library changes. Assume for now that they're test regressions as some assumptions are invalid, rather than library regressions. (I think you can see why we like to isolate SDK updates from everything else...)
Can you just do a checkout of trunk with no changes, run the same tests and see what the results are?
re-ran the scale tests with and without sdk update commit. I am seeing similar failures in both. Error samples included below.
I'll rebase this branch with #7021. I'll also look into ITestS3AConfiguration tests next.
without AWS SDK update:
command: mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale > scale_8_core_old_aws
[ERROR] testCostOfCreatingMagicFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost) Time elapsed: 2.922 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0004/test/testCostOfCreatingMagicFile/__magic_job-123/__base/file.txt.pending': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"]): Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"])
[ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob) Time elapsed: 37.374 s <<< FAILURE!
java.lang.AssertionError:
Job job_1725412925883_0003 failed in state FAILED with cause Job commit failed: org.apache.hadoop.fs.PathIOException: `s3a://s3atestingdsgrewal/job-00-fork-0008/test/ITestS3ACommitterMRJob-execute-magic/__magic_job-3cbf58ed-920a-48a7-9a68-2b888fb9ce11/job-3cbf58ed-920a-48a7-9a68-2b888fb9ce11/01/task_1725412925883_0003_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4270] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4270] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
[ERROR] Tests run: 5, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 20.563 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[ERROR] testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 5.037 s <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected expected:<kms+context> but was:<null>
[ERROR] testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 3.812 s <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
<[0, 1, 239, 4079]>
to contain exactly in any order:
<[0, 1, 255, 4095]>
elements not found:
<[255, 4095]>
and elements not expected:
<[239, 4079]>
[ERROR] testEncryptionEnabledAndDisabledFS(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 1.837 s <<< FAILURE!
java.lang.AssertionError: SecurityException should be thrown: "Exception should be raised if unencrypted data is read by a CSE enabled FS"
[ERROR] Tests run: 18, Failures: 0, Errors: 10, Skipped: 1, Time elapsed: 21.693 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[ERROR] testCentralEndpointAndNullRegionFipsWithCRUD(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion) Time elapsed: 2.665 s <<< ERROR!
java.lang.IllegalArgumentException: Non central endpoint cannot be set when fs.s3a.endpoint.fips is true : https://s3-accesspoint.us-east-1.amazonaws.com
[ERROR] testCentralEndpoint(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion) Time elapsed: 0.948 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid configuration: region from ARN `us-east-1` does not match client region `us-east-2` and UseArnRegion is `false`
[^[[1;31mERROR^[[m] testEndpointOverride(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion) Time elapsed: 0.998 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid configuration: region from ARN `us-east-1` does not match client region `us-east-2` and UseArnRegion is `false`
[ERROR] testCommitterWithDuplicatedCommit[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol) Time elapsed: 9.857 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0002/test/ITestMagicCommitProtocol-testCommitterWithDuplicatedCommit[track-commit-in-memory-false]/__magic_job-job_202409040534_0002/job-job_202409040534_0002/01/task_202409040534_0002_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4331] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4331] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
[ERROR] testMapFileOutputCommitter[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol) Time elapsed: 11.016 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0002/test/ITestMagicCommitProtocol-testMapFileOutputCommitter[track-commit-in-memory-false]/__magic_job-job_202409040503_0002/job-job_202409040503_0002/01/task_202409040503_0002_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 5604] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 5604] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
[ERROR] testComplexDirActions(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List) Time elapsed: 16.843 s <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>
[ERROR] testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename) Time elapsed: 4.976 s <<< FAILURE!
java.lang.AssertionError: Number of bytes copied: files_copied_bytes expected:<16384> but was:<16368>
[ERROR] testLargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp) Time elapsed: 5.72 s <<< ERROR!
java.io.FileNotFoundException: File file:/home/dsgrewal/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/ITestS3AContractDistCp/testLargeFilesFromRemote/local/outputDir/inputDir/file1 does not exist
[ERROR] test_110_teragen[magic-memory=false](org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A) Time elapsed: 20.265 s <<< FAILURE!
java.lang.AssertionError: teragen(1000, s3a://s3atestingdsgrewal/job-00/test/terasort-magic-false/sortin) failed expected:<0> but was:<1>
[ERROR] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics) Time elapsed: 2.27 s <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but was:<512>
[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass) Time elapsed: 0.572 s <<< FAILURE!
org.junit.ComparisonFailure: [List file status length S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/array/src/hugefile; isDirectory=false; length=10485744; replication=1; blocksize=33554432; modification_time=1725414230000; access_time=0; owner=dsgrewal; group=dsgrewal; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="8f247ca05cb4324d2bd3e1ec1a88e481-2" versionId=null] expected:<104857[60]L> but was:<104857[44]L>
With SDK update
command : dsgrewal@5671d360aa8a:~/hadoop/hadoop-tools/hadoop-aws$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale > scale_8_core_new_aws
error samples:
[ERROR] testCostOfCreatingMagicFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost) Time elapsed: 2.842 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0003/test/testCostOfCreatingMagicFile/__magic_job-123/__base/file.txt.pending': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"]): Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"])
[ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob) Time elapsed: 37.368 s <<< FAILURE!
java.lang.AssertionError:
Job job_1725417665118_0003 failed in state FAILED with cause Job commit failed: org.apache.hadoop.fs.PathIOException: `s3a://s3atestingdsgrewal/job-00-fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic_job-95d105a2-9a3d-4a55-b1d6-7cade7a2b755/job-95d105a2-9a3d-4a55-b1d6-7cade7a2b755/01/task_1725417665118_0003_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4267] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4267] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
[ERROR] testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 3.855 s <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
<[0, 1, 239, 4079]>
to contain exactly in any order:
<[0, 1, 255, 4095]>
elements not found:
<[255, 4095]>
and elements not expected:
<[239, 4079]>
[ERROR] testEncryptionEnabledAndDisabledFS(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 1.826 s <<< FAILURE!
java.lang.AssertionError: SecurityException should be thrown: "Exception should be raised if unencrypted data is read by a CSE enabled FS"
[ERROR] testEncryption(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms) Time elapsed: 1.657 s <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected expected:<kms+context> but was:<null>
[ERROR] testCentralEndpointAndNullRegionFipsWithCRUD(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion) Time elapsed: 2.916 s <<< ERROR!
java.lang.IllegalArgumentException: Non central endpoint cannot be set when fs.s3a.endpoint.fips is true : https://s3-accesspoint.us-east-1.amazonaws.com
[ERROR] testCentralEndpoint(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion) Time elapsed: 0.967 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid configuration: region from ARN `us-east-1` does not match client region `us-east-2` and UseArnRegion is `false`
[ERROR] testCommitterWithDuplicatedCommit[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol) Time elapsed: 9.582 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0007/test/ITestMagicCommitProtocol-testCommitterWithDuplicatedCommit[track-commit-in-memory-false]/__magic_job-job_202409040635_0007/job-job_202409040635_0007/01/task_202409040635_0007_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4326] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4326] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
[ERROR] testComplexDirActions(org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus) Time elapsed: 16.914 s <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>
[ERROR] testLargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp) Time elapsed: 5.542 s <<< ERROR!
java.io.FileNotFoundException: File file:/home/dsgrewal/hadoop/hadoop-tools/hadoop-aws/target/test-dir/7/ITestS3AContractDistCp/testLargeFilesFromRemote/local/outputDir/inputDir/file1 does not exist
[ERROR] test_110_teragen[magic-memory=false](org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A) Time elapsed: 20.716 s <<< FAILURE!
java.lang.AssertionError: teragen(1000, s3a://s3atestingdsgrewal/job-00/test/terasort-magic-false/sortin) failed expected:<0> but was:<1>
[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits) Time elapsed: 1.798 s <<< FAILURE!
org.junit.ComparisonFailure: [List file status length S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin; isDirectory=false; length=10485744; replication=1; blocksize=33554432; modification_time=1725418948000; access_time=0; owner=dsgrewal; group=dsgrewal; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="2c6748a4d4a12430c65b8f7d26665ae6-2" versionId=null] expected:<104857[60]L> but was:<104857[44]L>
[ERROR] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics) Time elapsed: 2.21 s <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but was:<512>
[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass) Time elapsed: 0.564 s <<< FAILURE!
org.junit.ComparisonFailure: [List file status length S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/array/src/hugefile; isDirectory=false; length=10485744; replication=1; blocksize=33554432; modification_time=1725418969000; access_time=0; owner=dsgrewal; group=dsgrewal; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="c6585c77653d07c9320355e92fef70d4-2" versionId=null] expected:<104857[60]L> but was:<104857[44]L>
BTW, @diljotgrewal does this new SDK fix https://github.com/aws/aws-sdk-java-v2/issues/5247
This is a serious problem and nobody seems be giving it attention. It also flags to me that there seems to be more attention in the library to new features over fundamental issues. It'd be nice if you can draw attention to the SDK team about this -the more internal pressure you can generate the more likely it is that someone may actually look at it.
This SDK is incomplete. It lets you set the flag on put/complete multipart upload, but not on CopyObjectRequest. We need this for rename. Given copy is just a PUT with a different source, I don't see why it shouldn't be available in S3 -so it's just an SDK omission
Am I correct?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 0s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 53s | Maven dependency ordering for branch | |
| -1 :x: | mvninstall | 33m 9s | /branch-mvninstall-root.txt | root in trunk failed. |
| +1 :green_heart: | compile | 17m 53s | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | compile | 16m 35s | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| -1 :x: | mvnsite | 12m 36s | /branch-mvnsite-root.txt | root in trunk failed. |
| -1 :x: | javadoc | 10m 7s | /branch-javadoc-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt | root in trunk failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javadoc | 8m 10s | /branch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt | root in trunk failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| -1 :x: | shadedclient | 52m 48s | branch has errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 1m 3s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 30m 43s | /patch-mvninstall-root.txt | root in the patch failed. |
| +1 :green_heart: | compile | 16m 59s | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 | |
| +1 :green_heart: | javac | 16m 59s | the patch passed | |
| +1 :green_heart: | compile | 16m 17s | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 | |
| +1 :green_heart: | javac | 16m 17s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -1 :x: | mvnsite | 6m 36s | /patch-mvnsite-root.txt | root in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | No new issues. | |
| -1 :x: | javadoc | 10m 32s | /patch-javadoc-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt | root in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javadoc | 8m 24s | /patch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt | root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| -1 :x: | shadedclient | 55m 21s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 756m 25s | /patch-unit-root.txt | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 44s | The patch does not generate ASF License warnings. | |
| 1037m 34s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7015 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux bbff94579452 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ebc8913c63182602cb718a8f666a2580c9685a56 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/3/testReport/ |
| Max. process+thread count | 4278 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/3/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Just talked to the SDK team, InstanceProfileCredentialsProvider issue is not fixed with this SDK release. It is on their radar and they plan to work on it soon.
Re: incomplete SDK: can you please give me some context? Which flag is missing in CopyObjectRequest?
Just talked to the SDK team, InstanceProfileCredentialsProvider issue is not fixed with this SDK release. It is on their radar and they plan to work on it soon.
happy
Re: incomplete SDK: can you please give me some context? Which flag is missing in CopyObjectRequest?
I couldn't see how to set the If-none-match header on a CopyObjectRequest
does this address the needless logging of HADOOP-19272? if not, later one is needed and you need to do the before/after checks to make sure things are better not worse -including a rename of a 128+ MB file. Think we should add a large file upload and for anyone testing in EC2 the upload has to go to a s3 store on another continent. if you can think of anything else like this, add it to testing.md as you do it. thanks
FYI, we fear SDK updates for subtle regressions like that. They can get past the automated testing but fill up the logs -which is why we absolutely require that step of manual review of all output files of the test run, manual exploratory testing etc. we're going to have to be even more rigorous now.
@diljotgrewal I've created a feature branch for this work: HADOOP-19256-s3-conditional-writes
Can you upgrade to the latest SDK, do the usual qualification work and then submit a request against this branch. I don't want to do any SDK updates to trunk itself until I can come up with a even more robust qualification. Which I think will have to include auditing bits of the SDK as well to see if there are new warnings added.
Shouldn't we target the latest one https://github.com/aws/aws-sdk-java-v2/tree/2.29.19 instead ? It comes with https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-express-one-zone-append-data-object/
You can do it you should be aware of that my process is going to be so rigourous it is going to cost you a couple of days. I intend to include using ripgrep Count the difference in ".warn(" strings in the bit of the code based we care about and whoever submit the PR is going to have to look at any increase identify the cause.
I estimate I have spent at least eight weeks this year dealing with SDK related problems. This means right now I'm completely indifferent to new features coming down as I don't even get the opportunity to actually write interesting code to make use of them. I don't think it is fair for other developers to be able to do that work deal with SDK problems
Now @diljotgrewal you can add a late version of the STK into the feature branch created for this work, but you at @shameersss1 I'm going to have to share the paint of qualifying the upgrade and fielding any regressions which come from it.
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}