hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-17649. Update wildfly openssl to 2.2.1.Final
trafficstars
Description of PR
Update wildfly openssl to 2.2.1.Final.
https://nvd.nist.gov/vuln/detail/CVE-2020-25644
A memory leak flaw was found in WildFly OpenSSL in versions prior to 1.1.3.Final, where it removes an HTTP session. It may allow the attacker to cause OOM leading to a denial of service. The highest threat from this vulnerability is to system availability.
JIRA - HADOOP-17649
How was this patch tested?
-
CI/Build.
-
After setting this property
<property>
<name>fs.s3a.ssl.channel.mode</name>
<value>openssl</value>
</property>
Added test results for hadoop-aws integration tests
For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [X] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestPaths
[INFO] Running org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.426 s - in org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.288 s - in org.apache.hadoop.fs.s3a.commit.staging.TestPaths
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestHeaderProcessing
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestCreateFileBuilder
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.38 s - in org.apache.hadoop.fs.s3a.impl.TestHeaderProcessing
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.46 s - in org.apache.hadoop.fs.s3a.impl.TestCreateFileBuilder
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestS3AMultipartUploaderSupport
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.367 s - in org.apache.hadoop.fs.s3a.impl.TestS3AMultipartUploaderSupport
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestRequestFactory
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestDirectoryMarkerPolicy
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.775 s - in org.apache.hadoop.fs.s3a.impl.TestRequestFactory
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.602 s - in org.apache.hadoop.fs.s3a.impl.TestDirectoryMarkerPolicy
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.377 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter
88665a380593:hadoop-aws ashugpt$
88665a380593:hadoop-aws ashugpt$
88665a380593:hadoop-aws ashugpt$ tail -f integration_test_response_hadoop_trunk.out
[INFO] Running org.apache.hadoop.fs.s3a.audit.TestLoggingAuditor
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.039 s - in org.apache.hadoop.fs.s3a.audit.TestHttpReferrerAuditHeader
[INFO] Running org.apache.hadoop.fs.s3a.TestS3ABlockOutputStream
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AUnbuffer
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.259 s - in org.apache.hadoop.fs.s3a.audit.TestLoggingAuditor
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestAuthoritativePath
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.062 s - in org.apache.hadoop.fs.s3a.s3guard.TestAuthoritativePath
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestMetastoreChecking
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.494 s - in org.apache.hadoop.fs.s3a.s3guard.TestMetastoreChecking
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 s - in org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.172 s - in org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.083 s - in org.apache.hadoop.fs.s3a.TestS3AUnbuffer
[INFO] Running org.apache.hadoop.fs.s3a.TestDataBlocks
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.822 s - in org.apache.hadoop.fs.s3a.TestDataBlocks
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.95 s - in org.apache.hadoop.fs.s3a.TestS3ABlockOutputStream
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.433 s - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
[INFO] Running org.apache.hadoop.fs.s3a.TestArnResource
[INFO] Running org.apache.hadoop.fs.s3native.TestS3xLoginHelper
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.114 s - in org.apache.hadoop.fs.s3a.TestArnResource
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.764 s - in org.apache.hadoop.fs.s3native.TestS3xLoginHelper
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.404 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.454 s - in org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.804 s - in org.apache.hadoop.fs.s3a.audit.impl.TestActiveAuditManagerThreadLeakage
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.458 s - in org.apache.hadoop.fs.s3a.TestS3AInputStreamRetry
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.71 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.766 s - in org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 406, Failures: 0, Errors: 0, Skipped: 4
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-aws ---
[INFO] Building jar: /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/hadoop-aws-3.4.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:copy-dependencies (default) @ hadoop-aws ---
...
...
...
...
[INFO]
[INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ hadoop-aws ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default-integration-test) @ hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.391 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 17, Time elapsed: 55.794 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[WARNING] Tests run: 16, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 106.574 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractContentSummary
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.799 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 116.207 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.144 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractContentSummary
[INFO] Running org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 23.567 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.191 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.746 s - in org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestAWSStatisticCollection
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.099 s - in org.apache.hadoop.fs.s3a.statistics.ITestAWSStatisticCollection
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.225 s - in org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.903 s - in org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.009 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 214.479 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 248.811 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
[INFO] Running org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 335.192 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.47 s - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[ERROR] Tests run: 14, Failures: 0, Errors: 1, Skipped: 4, Time elapsed: 213.419 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[ERROR] testSTS(org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials) Time elapsed: 11.626 s <<< ERROR!
com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462)
at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials.testSTS(ITestS3ATemporaryCredentials.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost
[ERROR] Tests run: 12, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 124.126 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[ERROR] test_200_execute[directory](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob) Time elapsed: 20.406 s <<< FAILURE!
java.lang.AssertionError:
Job job_1657332805778_0001 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-directory: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3445)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2336)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2355)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3443)
at org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter.setupJob(DirectoryStagingCommitter.java:74)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
... 14 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 35 more
.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:295)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR] test_200_execute[partitioned](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob) Time elapsed: 16.402 s <<< FAILURE!
java.lang.AssertionError:
Job job_1657332805778_0002 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-partitioned/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
at org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.setupJob(StagingCommitter.java:430)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 31 more
.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:295)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob) Time elapsed: 18.506 s <<< FAILURE!
java.lang.AssertionError:
Job job_1657332805778_0003 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-magic/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
at org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.setupJob(MagicS3GuardCommitter.java:104)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 31 more
.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:295)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestRenameDeleteRace
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.702 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.838 s - in org.apache.hadoop.fs.s3a.impl.ITestRenameDeleteRace
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestXAttrCost
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.266 s - in org.apache.hadoop.fs.s3a.impl.ITestXAttrCost
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[WARNING] Tests run: 26, Failures: 0, Errors: 0, Skipped: 26, Time elapsed: 57.353 s - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.312 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.384 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.199 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[WARNING] Tests run: 22, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 286.83 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 306.805 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.55 s - in org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 322.362 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestDowngradeSyncable
[WARNING] Tests run: 72, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 525.202 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.726 s - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.461 s - in org.apache.hadoop.fs.s3a.ITestDowngradeSyncable
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.763 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AMultipartUploadSizeLimits
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.936 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.811 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 17.822 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AMultipartUploadSizeLimits
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.534 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 22.98 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 8.885 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContentEncoding
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.191 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.869 s - in org.apache.hadoop.fs.s3a.ITestS3AContentEncoding
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.08 s - in org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestJceksIO
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 28.149 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.733 s - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestCustomSigner
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.137 s - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.639 s - in org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.782 s - in org.apache.hadoop.fs.s3a.auth.ITestCustomSigner
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.694 s - in org.apache.hadoop.fs.s3a.auth.ITestJceksIO
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.347 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 23.265 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 36.966 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 223.452 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.769 s - in org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 56.217 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[ERROR] testDelegationBindingMismatch2(org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem) Time elapsed: 3.096 s <<< ERROR!
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables: No AWS credentials
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:857)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:522)
at org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationIT.newS3AInstance(AbstractDelegationIT.java:103)
at org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem.testDelegationBindingMismatch2(ITestSessionDelegationInFileystem.java:511)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables: No AWS credentials
at org.apache.hadoop.fs.s3a.auth.MarshalledCredentials.validate(MarshalledCredentials.java:336)
at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.loadAWSCredentials(FullCredentialsTokenBinding.java:107)
at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.deployUnbonded(FullCredentialsTokenBinding.java:120)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.deployUnbonded(S3ADelegationTokens.java:251)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToAnyDelegationToken(S3ADelegationTokens.java:284)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.serviceStart(S3ADelegationTokens.java:205)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
... 19 more
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AStorageClass
[ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 2, Time elapsed: 28.99 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[ERROR] testCommonCrawlLookup[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob) Time elapsed: 0.107 s <<< ERROR!
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables: No AWS credentials
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:857)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:522)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3563)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3668)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3615)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:556)
at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:189)
at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:100)
at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.setup(ITestDelegatedMRJob.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables: No AWS credentials
at org.apache.hadoop.fs.s3a.auth.MarshalledCredentials.validate(MarshalledCredentials.java:336)
at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.loadAWSCredentials(FullCredentialsTokenBinding.java:107)
at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.deployUnbonded(FullCredentialsTokenBinding.java:120)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.deployUnbonded(S3ADelegationTokens.java:251)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToAnyDelegationToken(S3ADelegationTokens.java:284)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.serviceStart(S3ADelegationTokens.java:205)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
... 26 more
[ERROR] testJobSubmissionCollectsTokens[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob) Time elapsed: 0.099 s <<< ERROR!
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables: No AWS credentials
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:857)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:522)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3563)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3668)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3615)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:556)
at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:189)
at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:100)
at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.setup(ITestDelegatedMRJob.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables: No AWS credentials
at org.apache.hadoop.fs.s3a.auth.MarshalledCredentials.validate(MarshalledCredentials.java:336)
at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.loadAWSCredentials(FullCredentialsTokenBinding.java:107)
at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.deployUnbonded(FullCredentialsTokenBinding.java:120)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.deployUnbonded(S3ADelegationTokens.java:251)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToAnyDelegationToken(S3ADelegationTokens.java:284)
at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.serviceStart(S3ADelegationTokens.java:205)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
... 26 more
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[WARNING] Tests run: 25, Failures: 0, Errors: 0, Skipped: 25, Time elapsed: 76.937 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.447 s - in org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.626 s - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.658 s - in org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 16.433 s - in org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 57.47 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.659 s - in org.apache.hadoop.fs.s3a.ITestS3AStorageClass
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.127 s - in org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.379 s - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.832 s - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.934 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.826 s - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditAccessChecks
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.43 s - in org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditManager
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 199.136 s - in org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.782 s - in org.apache.hadoop.fs.s3a.audit.ITestAuditManager
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.541 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.733 s - in org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.126 s - in org.apache.hadoop.fs.s3a.audit.ITestAuditAccessChecks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.025 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 37.833 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.247 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.148 s - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.533 s - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.617 s - in org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.757 s - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 594.266 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.064 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.442 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 43.857 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AMkdirCost
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.704 s - in org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ARenameCost
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.862 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AMiscOperationCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.54 s - in org.apache.hadoop.fs.s3a.performance.ITestS3AMiscOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.96 s - in org.apache.hadoop.fs.s3a.performance.ITestS3AMkdirCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestCreateFileCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.41 s - in org.apache.hadoop.fs.s3a.performance.ITestS3ARenameCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 213.801 s - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.286 s - in org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.433 s - in org.apache.hadoop.fs.s3a.performance.ITestCreateFileCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.175 s - in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.026 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.142 s - in org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 8.87 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 283.275 s - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.511 s - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[WARNING] Tests run: 73, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 703.526 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 386.416 s - in org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.899 s - in org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 475.876 s - in org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestS3ACommitterMRJob.test_200_execute:295->Assert.fail:89 Job job_1657332805778_0001 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-directory: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3445)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2336)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2355)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3443)
at org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter.setupJob(DirectoryStagingCommitter.java:74)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
... 14 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 35 more
.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
[ERROR] ITestS3ACommitterMRJob.test_200_execute:295->Assert.fail:89 Job job_1657332805778_0003 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-magic/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
at org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.setupJob(MagicS3GuardCommitter.java:104)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 31 more
.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
[ERROR] ITestS3ACommitterMRJob.test_200_execute:295->Assert.fail:89 Job job_1657332805778_0002 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-partitioned/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
at org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.setupJob(StagingCommitter.java:430)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 31 more
.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
[ERROR] Errors:
[ERROR] ITestS3ATemporaryCredentials.testSTS:130 » SdkClient Unable to find a region v...
[ERROR] ITestDelegatedMRJob.setup:182->AbstractS3ATestBase.setup:100->AbstractFSContractTestBase.setup:189 » ServiceState
[ERROR] ITestDelegatedMRJob.setup:182->AbstractS3ATestBase.setup:100->AbstractFSContractTestBase.setup:189 » ServiceState
[ERROR] ITestSessionDelegationInFileystem.testDelegationBindingMismatch2:511->AbstractDelegationIT.newS3AInstance:103 » ServiceState
[INFO]
[ERROR] Tests run: 1128, Failures: 3, Errors: 4, Skipped: 190
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (sequential-integration-tests) @ hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.939 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestAggregateIOStatistics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.553 s - in org.apache.hadoop.fs.s3a.statistics.ITestAggregateIOStatistics
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.881 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed: 27.711 s - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.282 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerToolRootOperations
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.563 s - in org.apache.hadoop.fs.s3a.tools.ITestMarkerToolRootOperations
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.987 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.918 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.195 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.788 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.748 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 17.04 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 197.267 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 124, Failures: 0, Errors: 0, Skipped: 84
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @ hadoop-aws ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (default-integration-test) @ hadoop-aws ---
[ERROR] There are test failures.
Please refer to /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/failsafe-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (sequential-integration-tests) @ hadoop-aws ---
[ERROR] There are test failures.
Please refer to /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/failsafe-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29:09 min
[INFO] Finished at: 2022-07-09T03:36:38+01:00
[INFO] ------------------------------------------------------------------------
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 53s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 18s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 16s | trunk passed | |
| +1 :green_heart: | compile | 24m 53s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 21m 40s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | mvnsite | 19m 44s | trunk passed | |
| +1 :green_heart: | javadoc | 8m 31s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 7m 32s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | shadedclient | 38m 41s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 25m 46s | the patch passed | |
| +1 :green_heart: | compile | 24m 26s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 24m 26s | the patch passed | |
| +1 :green_heart: | compile | 21m 40s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 21m 40s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | mvnsite | 19m 15s | the patch passed | |
| +1 :green_heart: | shellcheck | 0m 1s | No new issues. | |
| +1 :green_heart: | javadoc | 8m 29s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 7m 26s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | shadedclient | 40m 27s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 1042m 48s | /patch-unit-root.txt | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 14s | The patch does not generate ASF License warnings. | |
| 1328m 32s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.security.ssl.TestDelegatingSSLSocketFactory |
| hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4539 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux 1866eb1dbc20 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cacf1454642a99a73af618dae775105fac336e48 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/1/testReport/ |
| Max. process+thread count | 2731 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@steveloughran - I saw your comments on this PR - https://github.com/apache/hadoop/pull/3029 about running the integration tests for
- hadoop-aws (Attached results in the PR)
- hadoop-azure - I dont have account with Azure, Can you suggest someone in community who can helop in running hadoop-azure tests for this upgrade?
Let me know if you have any comments for this upgrade.
also, be good to know what linux version you are testing on. i will have to test this on a vm before merging too, because the openssl native integration has been brittle in the past
I am testing this on:
System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
ok. i will do an (ec2) linux run later. that or fix my pi400 up for testing, which might be fun anyway
catching up on this. its' too late for 3.3.5, but we can get it into branch-3.3. for the successor. can you rebase?
catching up on this. its' too late for 3.3.5, but we can get it into branch-3.3. for the successor. can you rebase?
Hi @steveloughran , sorry got busy with other few important stuff earlier. I will rebase it.
Thanks for reminder.
catching up on this. its' too late for 3.3.5, but we can get it into branch-3.3. for the successor. can you rebase?
@steveloughran , I have rebased. Please help in review.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 56s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 3s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 33m 55s | trunk passed | |
| +1 :green_heart: | compile | 25m 23s | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | compile | 21m 54s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | mvnsite | 20m 10s | trunk passed | |
| -1 :x: | javadoc | 1m 22s | /branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 20s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | shadedclient | 34m 2s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 34m 19s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 31m 8s | the patch passed | |
| +1 :green_heart: | compile | 24m 40s | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | javac | 24m 40s | the patch passed | |
| +1 :green_heart: | compile | 21m 35s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 21m 35s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | mvnsite | 19m 49s | the patch passed | |
| +1 :green_heart: | shellcheck | 0m 0s | No new issues. | |
| -1 :x: | javadoc | 1m 11s | /patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | root in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 25s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | shadedclient | 34m 58s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 785m 11s | /patch-unit-root.txt | root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 1073m 7s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.security.ssl.TestDelegatingSSLSocketFactory |
| hadoop.hdfs.TestLeaseRecovery2 | |
| hadoop.hdfs.TestDFSInotifyEventInputStream |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4539 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux b35af670a7ef 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 217ff9474b7c8f43b5c95b7b030bd0f4dea73f94 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/2/testReport/ |
| Max. process+thread count | 2095 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/2/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
revisting this. we need
- a rebase
- full test of hadoop-azure abfs and aws s3a tests. these are the places which will break