hadoop
hadoop copied to clipboard
apache
HADOOP-18542. Keep MSI tenant ID and client ID optional
Description of PR
This PR is the same as https://github.com/apache/hadoop/pull/3788 but rebased onto a more recent version of the trunk branch.
This make tenant and client ID optional when getting an Azure token from the Azure Metadata Service since they can be inferred from the VM the calls is made from.
For some reason the integration tests were failing in @virajjasani 's version (couldn't get a useful error message) but once I rebased onto trunk it worked.
I ran the integration tests from an azure VM so that Azure's metadata service can infer the tenant and client ID (which is why they are optional in MsiTokenProvider)
How was this patch tested?
I ran the integration tests:
/hadoop/hadoop-tools/hadoop-azure# mvn -T 1C clean verify -Dtest=none -Dit.test=ITest*
[...]
[INFO] Results:
[INFO]
[WARNING] Tests run: 1559, Failures: 0, Errors: 0, Skipped: 1542
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (default) @ hadoop-azure ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36.325 s
[INFO] Finished at: 2022-05-06T15:23:32Z
[INFO] ------------------------------------------------------------------------
Let me know if something's missing, this is the first PR I do in this repo.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 18m 21s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 40m 21s | trunk passed | |
| +1 :green_heart: | compile | 0m 50s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 0m 45s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 0m 41s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 50s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 46s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 0m 40s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 1m 23s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 49s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed | |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 0m 38s | the patch passed | |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 0m 32s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 22s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 36s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 1m 9s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 27s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 0s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | |
| 120m 27s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 4d1618d485ba 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c451b246ee8e89a729b529fccfe90efbf4a298f7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/1/testReport/ |
| Max. process+thread count | 519 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Hi @pranavsaxena-microsoft, I've tried to follow this section of the documentation that you mentioned, but it's outdated:
- The folder
./src/test/resources/accountSettings/doesn't exist, nor the template file that I should use to create my account settings file dev-support/testrun-scripts/runtests.shshould prompt a menu, but in my case it runsAppendBlob-HNS-OAuthand gives me no choice
I followed the rest of the documentation by copying ./src/test/resources/azure-auth-keys.xml.template to ./src/test/resources/azure-auth-keys.xml and replacing those variables by the right values in the xml file:
{ABFS_ACCOUNT_NAME}{ACCOUNT_ACCESS_KEY}{TENANTID}{WASB_ACCOUNT_NAME}{WASB_FILESYSTEM}{CONTAINER_NAME}{ACCOUNT_NAME}
Then I ran
dev-support/testrun-scripts/runtests.sh -c "NonHNS-SharedKey"
[...]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.030 s
[INFO] Finished at: 2022-11-25T13:08:18Z
[INFO] ------------------------------------------------------------------------
Running the combination: NonHNS-SharedKey...
# Then it terminates without running anything
So it doesn't run the tests, same for HNS-OAuth and HNS-SharedKey. Do you know what I am missing for those tests to run ?
Hi @CLevasseur , the testing section requires you to have latest trunk commits into your branch (as the testing section has been updated for the trunk). Hence requesting you to kindly back-merge apache:trunk into your branch.
Thanks.
I have pulled the latest changes. It now fails when running the tests. Note that it also fails when I run the tests from the trunk branch of the apache/hadoop repository.
Also, it looks like those tests create a lot of containers in the storage account, is there an easy way to clean those ?
Tests Output (Both apache/hadoop:trunk and CLevasseur/hadoop:trunk give the same errors)
NonHNS-SharedKey:
Choose action:
[Note - SET_ACTIVE_TEST_CONFIG will help activate the config for IDE/single test class runs]
1) SET_ACTIVE_TEST_CONFIG
2) RUN_TEST
3) CLEAN_UP_OLD_TEST_CONTAINERS
4) SET_OR_CHANGE_TEST_ACCOUNT
5) PRINT_LOG4J_LOG_PATHS_FROM_LAST_RUN
#? 2
Enter parallel test run process count [default - 8]:
Set the active test combination to run the action:
1) HNS-OAuth
2) HNS-SharedKey
3) nonHNS-SharedKey
4) AppendBlob-HNS-OAuth
5) AllCombinationsTestRun
6) Quit
#? 3
Combination specific property setting: [ key=fs.azure.account.auth.type , value=SharedKey ]
Activated [src/test/resources/abfs-combination-test-configs.xml] - for account: **** for combination NonHNS-SharedKey
Running test for combination NonHNS-SharedKey on account **** [ProcessCount=8]
Test run report can be seen in dev-support/testlogs/2022-11-28_13-26-52/Test-Logs-NonHNS-SharedKey.txt
----- Test results -----
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[INFO]
[ERROR] Tests run: 567, Failures: 1, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob ? OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle ? OutputValidation `abfs:/...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter ? OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit ? OutputValidation `...
[INFO]
[ERROR] Tests run: 335, Failures: 1, Errors: 8, Skipped: 46
Time taken: 4 mins 51 secs.
Find test result for the combination (NonHNS-SharedKey) in: dev-support/testlogs/2022-11-28_13-26-52/Test-Logs-NonHNS-SharedKey.txt
Consolidated test result is saved in: dev-support/testlogs/2022-11-28_13-26-52/Test-Results.txt
------------------------
:::: AGGREGATED TEST RESULT ::::
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[INFO]
[ERROR] Tests run: 567, Failures: 1, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob ? OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle ? OutputValidation `abfs:/...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter ? OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit ? OutputValidation `...
[INFO]
[ERROR] Tests run: 335, Failures: 1, Errors: 8, Skipped: 46
And here are the detailed errors:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.001 s - in org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.398 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.006 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.074 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.343 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.599 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage
[ERROR] Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 32.453 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter
[ERROR] test_0420_validateJob(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter) Time elapsed: 1.076 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0001/test/TestJobThroughManifestCommitter/out%20put/dir-03-00/dir-02-00/dir-01-00/part-00-0033': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0001/test/TestJobThroughManifestCommitter/out put/_temporary/202211281329380545_0001/01/tasks/attempt_202211281329380545_0001_m_000000_1/dir-03-00/dir-02-00/dir-01-00/part-00-0033 with etag 0x8DAD144A33B175C and length 2 but found a file with etag 0x8DAD144A8F9F501 and length 2
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.939 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.493 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.575 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 7.903 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations
[WARNING] Tests run: 48, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 40.924 s - in org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.604 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 9.496 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.8 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.408 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.727 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.004 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen
[ERROR] Tests run: 17, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 25.873 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol
[ERROR] testCommitterWithDuplicatedCommit(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 11.446 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testCommitterWithDuplicatedCommit/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testCommitterWithDuplicatedCommit/_temporary/job_202211281330070044_0007/01/tasks/attempt_202211281330070044_0007_m_000000_0/part-m-00000 with etag 0x8DAD144AF3BAB0C and length 40 but found a file with etag 0x8DAD144B11B4823 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testMapFileOutputCommitter(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.774 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testMapFileOutputCommitter/part-m-00000/data': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testMapFileOutputCommitter/_temporary/job_202211281330190293_0007/01/tasks/attempt_202211281330190293_0007_m_000000_0/part-m-00000/data with etag 0x8DAD144B1D26F35 and length 296 but found a file with etag 0x8DAD144B2225839 and length 296
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend
[ERROR] testCommitLifecycle(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.42 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testCommitLifecycle/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testCommitLifecycle/_temporary/job_202211281330210232_0007/01/tasks/attempt_202211281330210232_0007_m_000000_0/part-m-00000 with etag 0x8DAD144B2D2C9A4 and length 40 but found a file with etag 0x8DAD144B31B87C4 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testConcurrentCommitTaskWithSubDir(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.3 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testConcurrentCommitTaskWithSubDir/SUB_DIR/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testConcurrentCommitTaskWithSubDir/_temporary/job_202211281330220759_0007/00/tasks/attempt_202211281330220759_0007_m_000000_0/SUB_DIR/part-m-00000 with etag 0x8DAD144B3BC1CF0 and length 40 but found a file with etag 0x8DAD144B3EA792E and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testParallelJobsToAdjacentPaths(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.218 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testParallelJobsToAdjacentPaths/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testParallelJobsToAdjacentPaths/_temporary/job_202211281330260574_0007/01/tasks/attempt_202211281330260574_0007_m_000000_0/part-m-00000 with etag 0x8DAD144B60171C7 and length 40 but found a file with etag 0x8DAD144B64A2FE2 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testOutputFormatIntegration(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.238 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testOutputFormatIntegration/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testOutputFormatIntegration/_temporary/job_202211281330290146_0007/01/tasks/attempt_202211281330290146_0007_m_000000_0/part-m-00000 with etag 0x8DAD144B79BABA6 and length 4 but found a file with etag 0x8DAD144B7EA3537 and length 4
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testTwoTaskAttemptsCommit(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.248 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testTwoTaskAttemptsCommit/attempt2-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0007/test/ITestAbfsManifestCommitProtocol-testTwoTaskAttemptsCommit/_temporary/job_202211281330310289_0007/01/tasks/attempt_202211281330310289_0007_m_000000_1/attempt2-m-00000 with etag 0x8DAD144B8F2A4C3 and length 40 but found a file with etag 0x8DAD144B923237B and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 10.631 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.253 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.532 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 9.738 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 8.85 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.156 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.564 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.53 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.619 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.313 s - in org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.9 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.008 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 99.707 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort
[ERROR] test_110_teragen(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time elapsed: 24.376 s <<< FAILURE!
java.lang.AssertionError: teragen(1000, abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.executeStage(ITestAbfsTerasort.java:211)
at org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.test_110_teragen(ITestAbfsTerasort.java:244)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
Any idea why those tests are failing ?
HNS is disabled on that storage account, so I can't run the configurations HNS-OAuth and HNS-SharedKey
I have created this JIRA to track this bug: https://issues.apache.org/jira/browse/HADOOP-18542
ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS looks related...what is the full stack? it may be a different exception came back from the azure call, or it actually worked.
other ones are not expected.
are you working with a hierarchical store here? as i think those tests are failures from that detail, not your change (essentially: etags aren't being preserved across rename).
do the same test suite against trunk and see if the tests still fail, if so, create a new jira (or maybe there is one already). it shouldn't block this pr
The exception for ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS is:
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 10, Time elapsed: 1.521 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] testCheckAccessForAccountWithoutNS(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.15 s <<< FAILURE!
java.lang.AssertionError: Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:529)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS(ITestAzureBlobFileSystemCheckAccess.java:181)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
are you working with a hierarchical store here? as i think those tests are failures from that detail, not your change (essentially: etags aren't being preserved across rename).
No, HNS is disabled on Azure's side, and in the account settings as well:
cat ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml | grep -B1 -A2 namespace
<property>
<name>fs.azure.test.namespace.enabled.****.dfs.core.windows.net</name>
<value>false</value>
</property>
<property>
<name>fs.azure.test.namespace.enabled</name>
<value>false</value>
</property>
do the same test suite against trunk and see if the tests still fail, if so, create a new jira (or maybe there is one already). it shouldn't block this pr
The result is the same, here are the logs I'm getting:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.976 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.174 s - in org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.921 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[ERROR] testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration) Time elapsed: 0.102 s <<< FAILURE!
java.lang.AssertionError: Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399)
at org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.396 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043 s - in org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.2 s - in org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.088 s - in org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.651 s - in org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.133 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.278 s - in org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.713 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.861 s - in org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.668 s - in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.598 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Running org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.322 s - in org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.61 s - in org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.534 s - in org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.441 s - in org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.849 s - in org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.786 s - in org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 6.956 s - in org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.59 s - in org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 s - in org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.686 s - in org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.877 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.459 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO]
[ERROR] There are test failures.
Please refer to /hadoop/hadoop-tools/hadoop-azure/target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar: /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ hadoop-azure ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (integration-test-abfs-parallel-classesAndMethods) @ hadoop-azure ---
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.jar (93 kB at 2.4 MB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.pom (7.3 kB at 292 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.pom (2.4 kB at 70 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.pom (2.6 kB at 146 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.jar (22 kB at 1.3 MB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.jar (40 kB at 1.8 MB/s)
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.294 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.288 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.765 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.222 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.588 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsPositionedRead
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.635 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsPositionedRead
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.634 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsUnbuffer
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.496 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsUnbuffer
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.503 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.634 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization
[WARNING] Tests run: 29, Failures: 0, Errors: 0, Skipped: 29, Time elapsed: 2.638 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestOauthOverAbfsScheme
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.277 s - in org.apache.hadoop.fs.azurebfs.ITestOauthOverAbfsScheme
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.317 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsDurationTrackers
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.97 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsDurationTrackers
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.973 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsClient
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 19.652 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl
[WARNING] Tests run: 66, Failures: 0, Errors: 0, Skipped: 56, Time elapsed: 11.749 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemProperties
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.013 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemProperties
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOauth
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.099 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOauth
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.549 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsOutputStreamStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.074 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsOutputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.745 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.404 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.549 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemStoreListStatusWithRange
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.767 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemStoreListStatusWithRange
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 10, Time elapsed: 1.521 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] testCheckAccessForAccountWithoutNS(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.15 s <<< FAILURE!
java.lang.AssertionError: Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:529)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS(ITestAzureBlobFileSystemCheckAccess.java:181)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.827 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.649 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.393 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMainOperation
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.373 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMainOperation
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFinalize
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.362 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFinalize
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.196 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.622 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsHugeFiles
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.34 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsHugeFiles
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.802 s - in org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestClientUrlScheme
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.604 s - in org.apache.hadoop.fs.azurebfs.ITestClientUrlScheme
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.775 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestSharedKeyAuth
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 s - in org.apache.hadoop.fs.azurebfs.ITestSharedKeyAuth
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.474 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestCustomerProvidedKey
[WARNING] Tests run: 28, Failures: 0, Errors: 0, Skipped: 28, Time elapsed: 0.882 s - in org.apache.hadoop.fs.azurebfs.ITestCustomerProvidedKey
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAttributes
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.402 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAttributes
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelegationSAS
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed: 0.41 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelegationSAS
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.298 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.921 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.37 s - in org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsListStatusRemoteIterator
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.053 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsListStatusRemoteIterator
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemPermission
[WARNING] Tests run: 128, Failures: 0, Errors: 0, Skipped: 128, Time elapsed: 9.17 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemPermission
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamSmallFileReads
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.622 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamSmallFileReads
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStream
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.918 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamReadFooter
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.717 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamReadFooter
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.544 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[INFO]
[ERROR] Tests run: 567, Failures: 1, Errors: 0, Skipped: 277
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.918 s - in org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.879 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.657 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.991 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.569 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.159 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.197 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO
[ERROR] Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 32.672 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter
[ERROR] test_0420_validateJob(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter) Time elapsed: 0.74 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0002/test/TestJobThroughManifestCommitter/out%20put/dir-03-00/dir-02-00/dir-01-00/part-00-0033': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0002/test/TestJobThroughManifestCommitter/out put/_temporary/202211291549220355_0002/01/tasks/attempt_202211291549220355_0002_m_000000_1/dir-03-00/dir-02-00/dir-01-00/part-00-0033 with etag 0x8DAD22153D4F458 and length 2 but found a file with etag 0x8DAD22159994F7E and length 2
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.371 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.86 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 8.559 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations
[WARNING] Tests run: 48, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 41.978 s - in org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.96 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 9.624 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.207 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer
[ERROR] Tests run: 17, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 25.661 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol
[ERROR] testCommitterWithDuplicatedCommit(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 10.143 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitterWithDuplicatedCommit/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitterWithDuplicatedCommit/_temporary/job_202211291549500881_0004/01/tasks/attempt_202211291549500881_0004_m_000000_0/part-m-00000 with etag 0x8DAD2215D8BA025 and length 40 but found a file with etag 0x8DAD2215F25DA02 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testMapFileOutputCommitter(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.857 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testMapFileOutputCommitter/part-m-00000/data': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testMapFileOutputCommitter/_temporary/job_202211291550000055_0004/01/tasks/attempt_202211291550000055_0004_m_000000_0/part-m-00000/data with etag 0x8DAD2215FF58EA2 and length 296 but found a file with etag 0x8DAD221603B8E0D and length 296
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testCommitLifecycle(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.462 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitLifecycle/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitLifecycle/_temporary/job_202211291550020019_0004/01/tasks/attempt_202211291550020019_0004_m_000000_0/part-m-00000 with etag 0x8DAD22160F41493 and length 40 but found a file with etag 0x8DAD22161466E35 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testConcurrentCommitTaskWithSubDir(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.262 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testConcurrentCommitTaskWithSubDir/SUB_DIR/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testConcurrentCommitTaskWithSubDir/_temporary/job_202211291550030237_0004/00/tasks/attempt_202211291550030237_0004_m_000000_1/SUB_DIR/part-m-00000 with etag 0x8DAD22161DB6C66 and length 40 but found a file with etag 0x8DAD22162100941 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testParallelJobsToAdjacentPaths(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.401 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testParallelJobsToAdjacentPaths/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testParallelJobsToAdjacentPaths/_temporary/job_202211291550080667_0004/01/tasks/attempt_202211291550080667_0004_m_000000_0/part-m-00000 with etag 0x8DAD221646E60AB and length 40 but found a file with etag 0x8DAD22164B4AE25 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testOutputFormatIntegration(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 0.925 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testOutputFormatIntegration/part-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testOutputFormatIntegration/_temporary/job_202211291550110055_0004/01/tasks/attempt_202211291550110055_0004_m_000000_0/part-m-00000 with etag 0x8DAD221662398A8 and length 4 but found a file with etag 0x8DAD221664D1399 and length 4
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testTwoTaskAttemptsCommit(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) Time elapsed: 1.291 s <<< ERROR!
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testTwoTaskAttemptsCommit/attempt2-m-00000': Expected the file renamed from abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testTwoTaskAttemptsCommit/_temporary/job_202211291550130427_0004/01/tasks/attempt_202211291550130427_0004_m_000000_1/attempt2-m-00000 with etag 0x8DAD221676CB12D and length 40 but found a file with etag 0x8DAD22167ACBE08 and length 40
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166)
at org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.58 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.177 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.624 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.35 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.322 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.125 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 6.888 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 9.429 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.189 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.352 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.206 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.014 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.823 s - in org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.969 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.734 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 100.527 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort
[ERROR] test_110_teragen(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time elapsed: 25.881 s <<< FAILURE!
java.lang.AssertionError: teragen(1000, abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.executeStage(ITestAbfsTerasort.java:211)
at org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.test_110_teragen(ITestAbfsTerasort.java:244)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob ? OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle ? OutputValidation `abfs:/...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter ? OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths ? OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit ? OutputValidation `...
[INFO]
[ERROR] Tests run: 335, Failures: 1, Errors: 8, Skipped: 46
I'll create another JIRA to fix those
Here is the JIRA to track those integration test failures: https://issues.apache.org/jira/browse/HADOOP-18545
Hi,
ITestAbfsTerasort.test_110_teragen, ITestAbfsJobThroughManifestCommitter.test_0420_validateJob, ITestAbfsManifestCommitProtocol failures are expected as of now.
ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS failure is not expected.
As your changes are around OAuth, you would need an HNS account for the testing.
Thanks.
Hi, sorry for the delay, it took me a while to set everything up. Unfortunately I am still getting issues when running the integration tests on the apache/hadoop repo (i'm not even using my fork at this point).
It looks like permission issues, although the roles look correct to me. I followed the doc asking for:
3. A Service principal with RBAC - Storage Blob Data Owner (superuser) role on account
4. A Service principal with no RBAC roles assigned on account. This is needed for checkaccess tests (noRBACUser)
5. A Service principal with Storage Blob Data Contributor RBAC roles assigned on account. This is needed for RBAC tests (contributorRBACUser)
6. A Service principal with Storage Blob Data Reader RBAC roles assigned on account. This is needed for RBAC tests (readerRBACUser)
The credentials of each of these 4 account was set in the configuration file.
Here is the output of running the HNS-OAuth test suite, can you tell me if those failures are expected, and if not, what I might miss ?
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.392 s - in org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.225 s - in org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.192 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[ERROR] testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration) Time elapsed: 0.19 s <<< FAILURE!
java.lang.AssertionError: Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399)
at org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.404 s - in org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.809 s - in org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.516 s - in org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.182 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.515 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.529 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.637 s - in org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.092 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 s - in org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.03 s - in org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Running org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 s - in org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.304 s - in org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.894 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.627 s - in org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.953 s - in org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.236 s - in org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.319 s - in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.355 s - in org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Running org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.834 s - in org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.102 s - in org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.466 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.788 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 99.715 s - in org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 111, Failures: 1, Errors: 0, Skipped: 3
[INFO]
[ERROR] There are test failures.
Please refer to /hadoop/hadoop-tools/hadoop-azure/target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar: /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ hadoop-azure ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- cyclonedx-maven-plugin:2.7.3:makeBom (default) @ hadoop-azure ---
[INFO] CycloneDX: Parameters
[INFO] ------------------------------------------------------------------------
[INFO] schemaVersion : 1.4
[INFO] includeBomSerialNumber : true
[INFO] includeCompileScope : true
[INFO] includeProvidedScope : true
[INFO] includeRuntimeScope : true
[INFO] includeTestScope : false
[INFO] includeSystemScope : true
[INFO] includeLicenseText : false
[INFO] outputReactorProjects : true
[INFO] outputFormat : all
[INFO] outputName : bom
[INFO] ------------------------------------------------------------------------
[INFO] CycloneDX: Resolving Dependencies
[INFO] CycloneDX: Creating BOM
[INFO] CycloneDX: Writing BOM (XML): /hadoop/hadoop-tools/hadoop-azure/target/bom.xml
[INFO] CycloneDX: Validating BOM (XML): /hadoop/hadoop-tools/hadoop-azure/target/bom.xml
[INFO] CycloneDX: Writing BOM (JSON): /hadoop/hadoop-tools/hadoop-azure/target/bom.json
[INFO] CycloneDX: Validating BOM (JSON): /hadoop/hadoop-tools/hadoop-azure/target/bom.json
[WARNING] Unknown keyword additionalItems - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (integration-test-abfs-parallel-classesAndMethods) @ hadoop-azure ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.066 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestSharedKeyAuth
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.615 s - in org.apache.hadoop.fs.azurebfs.ITestSharedKeyAuth
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestOauthOverAbfsScheme
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.291 s - in org.apache.hadoop.fs.azurebfs.ITestOauthOverAbfsScheme
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.38 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.184 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.445 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsHugeFiles
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.457 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsHugeFiles
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.489 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFinalize
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.99 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFinalize
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOauth
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.77 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOauth
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.414 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.598 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMainOperation
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemStoreListStatusWithRange
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.689 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMainOperation
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.372 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemStoreListStatusWithRange
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 1.001 s - in org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsUnbuffer
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.873 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsUnbuffer
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.144 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsPositionedRead
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.692 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsPositionedRead
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.391 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl
[WARNING] Tests run: 66, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 10.094 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.707 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.933 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestClientUrlScheme
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.419 s - in org.apache.hadoop.fs.azurebfs.ITestClientUrlScheme
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 5.866 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.084 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.978 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.116 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.784 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemPermission
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.319 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[INFO] Tests run: 128, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.366 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemPermission
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsListStatusRemoteIterator
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAttributes
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.988 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAttributes
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.248 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.267 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsListStatusRemoteIterator
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemProperties
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.015 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemProperties
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.557 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.521 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestCustomerProvidedKey
[WARNING] Tests run: 28, Failures: 0, Errors: 0, Skipped: 28, Time elapsed: 0.769 s - in org.apache.hadoop.fs.azurebfs.ITestCustomerProvidedKey
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] Tests run: 12, Failures: 6, Errors: 1, Skipped: 1, Time elapsed: 2.714 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] testFsActionWRITE(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 2.111 s <<< FAILURE!
java.lang.AssertionError: Should have been given access WRITE on abfss://abfs-testcontainer-4d803dd2-6863-4838-b9d2-45a8b4f5fe85@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolder55f27a0ad7c1/test5.txt
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.assertAccessible(ITestAzureBlobFileSystemCheckAccess.java:326)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testFsActionWRITE(ITestAzureBlobFileSystemCheckAccess.java:252)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testFsActionREAD(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 2.082 s <<< FAILURE!
java.lang.AssertionError: Should have been given access READ on abfss://abfs-testcontainer-f32659ec-61fb-4ec3-865b-4dbbb4c9511c@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolderf131b527a2a5/test4.txt
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.assertAccessible(ITestAzureBlobFileSystemCheckAccess.java:326)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testFsActionREAD(ITestAzureBlobFileSystemCheckAccess.java:237)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testCheckAccessForNonExistentFile(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.914 s <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.io.FileNotFoundException> but was<org.apache.hadoop.security.AccessControlException>
at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:30)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.hadoop.security.AccessControlException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, HEAD, https://hadoopazuretests.dfs.core.windows.net/abfs-testcontainer-4a2b1026-c6c5-4e89-9f12-1e78c6161dd1/fork-0005/test/CheckAccessTestFolder6b7a4c22ebc4/nonExistentFile1.txt?action=checkAccess&fsAction=r--&timeout=90
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkCheckAccessException(AzureBlobFileSystem.java:1449)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.access(AzureBlobFileSystem.java:1267)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testCheckAccessForNonExistentFile(ITestAzureBlobFileSystemCheckAccess.java:131)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
... 7 more
Caused by: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, HEAD, https://hadoopazuretests.dfs.core.windows.net/abfs-testcontainer-4a2b1026-c6c5-4e89-9f12-1e78c6161dd1/fork-0005/test/CheckAccessTestFolder6b7a4c22ebc4/nonExistentFile1.txt?action=checkAccess&fsAction=r--&timeout=90
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:234)
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:194)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:192)
at org.apache.hadoop.fs.azurebfs.services.AbfsClient.checkAccess(AbfsClient.java:1076)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.access(AzureBlobFileSystemStore.java:1548)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.access(AzureBlobFileSystem.java:1265)
... 17 more
[ERROR] testFsActionEXECUTE(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.88 s <<< FAILURE!
java.lang.AssertionError: Should have been given access EXECUTE on abfss://abfs-testcontainer-0f7bb879-469d-499c-9949-c12f8a29e7b1@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFoldereec6282e78a9/test3.txt
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.assertAccessible(ITestAzureBlobFileSystemCheckAccess.java:326)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testFsActionEXECUTE(ITestAzureBlobFileSystemCheckAccess.java:222)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testFsActionREADEXECUTE(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.647 s <<< FAILURE!
java.lang.AssertionError: Should have been given access EXECUTE on abfss://abfs-testcontainer-c6593d53-53af-472f-87b3-3aebcf49ad2f@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFoldere8852e291d68/test6.txt
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.assertAccessible(ITestAzureBlobFileSystemCheckAccess.java:326)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testFsActionREADEXECUTE(ITestAzureBlobFileSystemCheckAccess.java:267)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testFsActionWRITEEXECUTE(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 2.182 s <<< FAILURE!
java.lang.AssertionError: Should have been given access EXECUTE on abfss://abfs-testcontainer-423cd975-0340-4eed-b96b-6e2dee420fcf@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolderc7c7a77c8539/test7.txt
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.assertAccessible(ITestAzureBlobFileSystemCheckAccess.java:326)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testFsActionWRITEEXECUTE(ITestAzureBlobFileSystemCheckAccess.java:282)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] testFsActionALL(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.887 s <<< FAILURE!
java.lang.AssertionError: Should have been given access EXECUTE on abfss://abfs-testcontainer-098e6633-262e-4bf0-bad5-f5ab600f2d55@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolderb1c428a49d24/test8.txt
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.assertAccessible(ITestAzureBlobFileSystemCheckAccess.java:326)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testFsActionALL(ITestAzureBlobFileSystemCheckAccess.java:297)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelegationSAS
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed: 0.405 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelegationSAS
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsClient
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 53.346 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.675 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsOutputStreamStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.041 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsOutputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.7 s - in org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization
[WARNING] Tests run: 29, Failures: 0, Errors: 0, Skipped: 29, Time elapsed: 0.911 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsDurationTrackers
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.83 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsDurationTrackers
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStream
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.76 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.314 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamSmallFileReads
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.62 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamSmallFileReads
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamReadFooter
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.675 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamReadFooter
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.806 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemCheckAccess.testFsActionALL:297->assertAccessible:326->Assert.assertTrue:42->Assert.fail:89 Should have been given access EXECUTE on abfss://abfs-testcontainer-098e6633-262e-4bf0-bad5-f5ab600f2d55@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolderb1c428a49d24/test8.txt
[ERROR] ITestAzureBlobFileSystemCheckAccess.testFsActionEXECUTE:222->assertAccessible:326->Assert.assertTrue:42->Assert.fail:89 Should have been given access EXECUTE on abfss://abfs-testcontainer-0f7bb879-469d-499c-9949-c12f8a29e7b1@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFoldereec6282e78a9/test3.txt
[ERROR] ITestAzureBlobFileSystemCheckAccess.testFsActionREAD:237->assertAccessible:326->Assert.assertTrue:42->Assert.fail:89 Should have been given access READ on abfss://abfs-testcontainer-f32659ec-61fb-4ec3-865b-4dbbb4c9511c@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolderf131b527a2a5/test4.txt
[ERROR] ITestAzureBlobFileSystemCheckAccess.testFsActionREADEXECUTE:267->assertAccessible:326->Assert.assertTrue:42->Assert.fail:89 Should have been given access EXECUTE on abfss://abfs-testcontainer-c6593d53-53af-472f-87b3-3aebcf49ad2f@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFoldere8852e291d68/test6.txt
[ERROR] ITestAzureBlobFileSystemCheckAccess.testFsActionWRITE:252->assertAccessible:326->Assert.assertTrue:42->Assert.fail:89 Should have been given access WRITE on abfss://abfs-testcontainer-4d803dd2-6863-4838-b9d2-45a8b4f5fe85@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolder55f27a0ad7c1/test5.txt
[ERROR] ITestAzureBlobFileSystemCheckAccess.testFsActionWRITEEXECUTE:282->assertAccessible:326->Assert.assertTrue:42->Assert.fail:89 Should have been given access EXECUTE on abfss://abfs-testcontainer-423cd975-0340-4eed-b96b-6e2dee420fcf@hadoopazuretests.dfs.core.windows.net/fork-0005/test/CheckAccessTestFolderc7c7a77c8539/test7.txt
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForNonExistentFile ? Unexp...
[INFO]
[ERROR] Tests run: 568, Failures: 6, Errors: 1, Skipped: 99
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.718 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.162 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 14.111 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.451 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.735 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort
[INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.123 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.632 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.903 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.482 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter
[INFO] Running org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.634 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.48 s - in org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.85 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename
[WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 12, Time elapsed: 2.211 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.279 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.882 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.83 s - in org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.566 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.106 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.877 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.391 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.13 s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.265 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.587 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 9.375 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory
[INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 7.908 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.409 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.828 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 10.293 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.084 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.628 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics
[WARNING] Tests run: 48, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 43.821 s - in org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 99.383 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort
[ERROR] test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time elapsed: 0.965 s <<< ERROR!
java.io.IOException: The ownership on the staging directory /hadoop/hadoop-tools/hadoop-azure/target/test-dir/6/yarn-2023-01-27-12.46.30.90/yarn-47034590/apps_staging_dir/root/.staging is not as expected. It is owned by 89e99dea-dc7e-48c8-8e8c-a41a97e6b877. The directory must be owned by the submitter root or root
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:152)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:147)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1677)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1674)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1674)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1695)
at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:334)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.executeStage(ITestAbfsTerasort.java:206)
at org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.test_120_terasort(ITestAbfsTerasort.java:262)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 ? IO The ownership o...
[INFO]
[ERROR] Tests run: 335, Failures: 0, Errors: 1, Skipped: 54
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (integration-test-abfs-parallel-classesAndMethods) @ hadoop-azure ---
[ERROR] There are test failures.
Please refer to /hadoop/hadoop-tools/hadoop-azure/target/failsafe-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[ERROR] There are test failures.
Please refer to /hadoop/hadoop-tools/hadoop-azure/target/failsafe-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:24 min (Wall Clock)
[INFO] Finished at: 2023-01-27T12:48:04Z
[INFO] ------------------------------------------------------------------------
that test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort failure is happening because yarn wants the owner of a file as determined from getFileStatus to match the yarn user (here: the username of the person running the test)
Look at IdentityTransformer and its configuration keys to see what you need to set for a mapping
I have had a look at IdentityTransformer (we're talking about those configuration keys, right ?)but I am not sure what those fields are for and how to set them, I have tried the following snippet but it failed with similar errors.
<property>
<name>fs.azure.identity.transformer.domain.name</name>
<value>$superuser</value>
</property>
<property>
<name>fs.azure.identity.transformer.service.principal.substitution.list</name>
<value>*</value>
</property>
I have added a unit-test to check that we can now instantiate an MsiTokenProvider without setting the client and tenant IDs in the configuration. The integration tests that are failing look unrelated to my changes (see test run in my previous comment).
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 46s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 48m 3s | trunk passed | |
| +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 39s | trunk passed | |
| -1 :x: | javadoc | 0m 35s | /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 10s | trunk passed | |
| +1 :green_heart: | shadedclient | 26m 34s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 36s | the patch passed | |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | javac | 0m 32s | the patch passed | |
| +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 27s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed | |
| -1 :x: | javadoc | 0m 23s | /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 4s | the patch passed | |
| +1 :green_heart: | shadedclient | 26m 14s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 1m 53s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | |
| 113m 30s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ae810286bc1d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6e6606c3f16770ed0e2ad8035d849cb5eaa2103d |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/2/testReport/ |
| Max. process+thread count | 585 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 47m 33s | trunk passed | |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 38s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed | |
| -1 :x: | javadoc | 0m 41s | /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 35s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 18s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 10s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 37s | the patch passed | |
| +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | javac | 0m 37s | the patch passed | |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 30s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 36s | the patch passed | |
| -1 :x: | javadoc | 0m 25s | /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 10s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 3s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 15s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | |
| 108m 4s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 325f9ba8fe4a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 50a67fe0a469a8ab50b9519d19ff0810b1ce3c3b |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/4/testReport/ |
| Max. process+thread count | 636 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 26s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 49m 48s | trunk passed | |
| +1 :green_heart: | compile | 0m 44s | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 34s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 46s | trunk passed | |
| -1 :x: | javadoc | 0m 42s | /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 14s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 21s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 39s | the patch passed | |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | javac | 0m 34s | the patch passed | |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 32s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 21s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 34s | the patch passed | |
| -1 :x: | javadoc | 0m 26s | /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 9s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 22s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 15s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | |
| 111m 33s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ba038e9aa362 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 50a67fe0a469a8ab50b9519d19ff0810b1ce3c3b |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/3/testReport/ |
| Max. process+thread count | 692 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 48m 46s | trunk passed | |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 35s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed | |
| -1 :x: | javadoc | 0m 41s | /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 34s | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 12s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 30s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 38s | the patch passed | |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 | |
| +1 :green_heart: | javac | 0m 34s | the patch passed | |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 30s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed | |
| -1 :x: | javadoc | 0m 25s | /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt | hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 4s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 15s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 10s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | |
| 109m 24s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e42f4c9fa1c2 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 50a67fe0a469a8ab50b9519d19ff0810b1ce3c3b |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/5/testReport/ |
| Max. process+thread count | 561 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@pranavsaxena-microsoft thoughts? would you be able to download this PR and test locally?
the test which is a definite regression is
ERROR] testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration) Time elapsed: 0.19 s <<< FAILURE!
java.lang.AssertionError: Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399)
at org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
...that is a test which came in with the mandatoryPassword code. you will have to edit CONFIG_KEYS to remove the keys which are now optional.
as to the others, I don't think they are related, more that you are testing through a non-HNS a/c and too much of the test code has drifted to being HNS-only.
push up a patch for the CONFIG_KEYS change and I will download and test; if it works for me I'll be happy
I just reran the whole test suite on a fresh install, and this time everything passed, without changing the code 🤷
Note that I had switched to an HNS account last time already.
The unit test you mention was failing on the implementation for the ClientCredsTokenProvider, which I hadn't changed (I only changed the logic for MsiTokenProvider), and was also failing when running from the upstream trunk branch.
There might have been some build cache still using a broken version, it's kind of a mystery to me to be honest.
Here is the test output, feel free to run it yourself:
/hadoop/hadoop-tools/hadoop-azure# dev-support/testrun-scripts/runtests.sh
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: linux
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 5.15
[INFO] os.detected.version.major: 5
[INFO] os.detected.version.minor: 15
[INFO] os.detected.release: ubuntu
[INFO] os.detected.release.version: 22.04
[INFO] os.detected.release.like.ubuntu: true
[INFO] os.detected.release.like.debian: true
[INFO] os.detected.classifier: linux-x86_64
[INFO]
[INFO] -------------------< org.apache.hadoop:hadoop-azure >-------------------
[INFO] Building Apache Hadoop Azure support 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting /hadoop/hadoop-tools/hadoop-azure/target
[INFO] Deleting /hadoop/hadoop-tools/hadoop-azure (includes = [dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir: /hadoop/hadoop-tools/hadoop-azure/target/test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (banned-illegal-imports) @ hadoop-azure ---
[INFO]
[INFO] --- replacer:1.5.3:replace (replace-generated-sources) @ hadoop-azure ---
[INFO] Skipping
[INFO]
[INFO] --- replacer:1.5.3:replace (replace-sources) @ hadoop-azure ---
[INFO] Skipping
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure ---
[INFO] Compiling 197 source files to /hadoop/hadoop-tools/hadoop-azure/target/classes
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java: Some input files use or override a deprecated API.
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java: Recompile with -Xlint:deprecation for details.
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: /hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java uses unchecked or unsafe operations.
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:list (deplist) @ hadoop-azure ---
[INFO] Can't extract module name from hadoop-shaded-protobuf_3_7-1.1.1.jar: hadoop.shaded.protobuf.3.7: Invalid module name: '3' is not a Java identifier
[INFO] Can't extract module name from geronimo-jcache_1.0_spec-1.0-alpha-1.jar: geronimo.jcache.1.0.spec: Invalid module name: '1' is not a Java identifier
[INFO]
[INFO] --- replacer:1.5.3:replace (replace-test-sources) @ hadoop-azure ---
[INFO] Skipping
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 14 resources
[INFO]
[INFO] --- replacer:1.5.3:replace (replace-generated-test-sources) @ hadoop-azure ---
[INFO] Skipping
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-azure ---
[INFO] Compiling 219 source files to /hadoop/hadoop-tools/hadoop-azure/target/test-classes
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestWithTimeout.java: Some input files use or override a deprecated API.
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestWithTimeout.java: Recompile with -Xlint:deprecation for details.
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/ClassicDelegationTokenManager.java: Some input files use unchecked or unsafe operations.
[WARNING] /hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/ClassicDelegationTokenManager.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M1:test (default-test) @ hadoop-azure ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO] Building jar: /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ hadoop-azure ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar: /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default) @ hadoop-azure ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (default) @ hadoop-azure ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ hadoop-azure ---
[INFO] Installing /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT.jar to /root/.m2/repository/org/apache/hadoop/hadoop-azure/3.4.0-SNAPSHOT/hadoop-azure-3.4.0-SNAPSHOT.jar
[INFO] Installing /hadoop/hadoop-tools/hadoop-azure/pom.xml to /root/.m2/repository/org/apache/hadoop/hadoop-azure/3.4.0-SNAPSHOT/hadoop-azure-3.4.0-SNAPSHOT.pom
[INFO] Installing /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT-tests.jar to /root/.m2/repository/org/apache/hadoop/hadoop-azure/3.4.0-SNAPSHOT/hadoop-azure-3.4.0-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.235 s
[INFO] Finished at: 2023-02-10T10:28:58Z
[INFO] ------------------------------------------------------------------------
Running the combination: AppendBlob-HNS-OAuth...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 107, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO]
[WARNING] Tests run: 331, Failures: 0, Errors: 0, Skipped: 331
Time taken: 1 mins 17 secs.
Find test logs for the combination (AppendBlob-HNS-OAuth) in: dev-support/testlogs/2023-02-10_10-28-59/Test-Logs-AppendBlob-HNS-OAuth.txt
Find consolidated test results in: dev-support/testlogs/2023-02-10_10-28-59/Test-Results.txt
----------
Running the combination: HNS-OAuth...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 107, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO]
[WARNING] Tests run: 331, Failures: 0, Errors: 0, Skipped: 331
Time taken: 1 mins 18 secs.
Find test logs for the combination (HNS-OAuth) in: dev-support/testlogs/2023-02-10_10-28-59/Test-Logs-HNS-OAuth.txt
Find consolidated test results in: dev-support/testlogs/2023-02-10_10-28-59/Test-Results.txt
----------
Running the combination: HNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 107, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO]
[WARNING] Tests run: 331, Failures: 0, Errors: 0, Skipped: 331
Time taken: 1 mins 18 secs.
Find test logs for the combination (HNS-SharedKey) in: dev-support/testlogs/2023-02-10_10-28-59/Test-Logs-HNS-SharedKey.txt
Find consolidated test results in: dev-support/testlogs/2023-02-10_10-28-59/Test-Results.txt
----------
Running the combination: NonHNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 107, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO]
[WARNING] Tests run: 331, Failures: 0, Errors: 0, Skipped: 331
Time taken: 1 mins 17 secs.
Find test logs for the combination (NonHNS-SharedKey) in: dev-support/testlogs/2023-02-10_10-28-59/Test-Logs-NonHNS-SharedKey.txt
Find consolidated test results in: dev-support/testlogs/2023-02-10_10-28-59/Test-Results.txt
----------
that test looks like it is failing because it was expecting a failure and now you've made an option optional, it isn't
private static final List<String> CONFIG_KEYS =
Collections.unmodifiableList(Arrays.asList(
FS_AZURE_ACCOUNT_OAUTH_CLIENT_ENDPOINT,
FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID,
FS_AZURE_ACCOUNT_OAUTH_CLIENT_SECRET));
now, maybe it is working for you because your auth-keys xml file includes different things. why not try commenting them out locally and running the unit tests (not the iTests) and see what the outcome is
That test is working when just running from a fresh install, without changing anything to the auth-keys xml file:
mvn test -Dtest=TestAccountConfiguration#testConfigPropNotFound
[INFO] Scanning for projects...
[...]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.36 s - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.540 s
[INFO] Finished at: 2023-03-10T12:27:13+01:00
[INFO] ------------------------------------------------------------------------
I am not sure what you are asking for ? The code this test covers is unrelated to my changes, and is already half broken on the upstream trunk branch. Should I just remove that test ?
Here is the part of the code covered by this test, while my changes impact this code instead.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 48s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 41m 34s | trunk passed | |
| +1 :green_heart: | compile | 0m 39s | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 39s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 37s | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | spotbugs | 1m 15s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 55s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed | |
| +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javac | 0m 33s | the patch passed | |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | javac | 0m 28s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | spotbugs | 1m 4s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 14s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 1m 56s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | |
| 102m 11s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fe4472a70cad 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b63e263532cbfb9808efb9e6154902b7315f7158 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/6/testReport/ |
| Max. process+thread count | 579 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 13m 2s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 38m 59s | trunk passed | |
| +1 :green_heart: | compile | 0m 39s | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | checkstyle | 0m 32s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 38s | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | spotbugs | 1m 21s | trunk passed | |
| +1 :green_heart: | shadedclient | 20m 50s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed | |
| +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javac | 0m 33s | the patch passed | |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | javac | 0m 28s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 36s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | spotbugs | 1m 7s | the patch passed | |
| +1 :green_heart: | shadedclient | 20m 22s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 16s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | |
| 106m 43s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a3980aaecdc6 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fb5fa8339029b667860a3cc52311efa5c8da659b |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/7/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 46s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 39m 52s | trunk passed | |
| +1 :green_heart: | compile | 0m 44s | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | checkstyle | 0m 34s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 38s | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | spotbugs | 1m 15s | trunk passed | |
| +1 :green_heart: | shadedclient | 20m 11s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed | |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javac | 0m 34s | the patch passed | |
| +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | javac | 0m 29s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed | |
| +1 :green_heart: | mvnsite | 0m 33s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | |
| +1 :green_heart: | spotbugs | 1m 6s | the patch passed | |
| +1 :green_heart: | shadedclient | 20m 25s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 18s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | |
| 94m 49s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4262 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 260bec46b8bc 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6501eaed6daa0888b1180c1dad456d43c3e1927b |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/8/testReport/ |
| Max. process+thread count | 721 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4262/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
I've seen this has been neglected for a while.
@saxenapranav and @anujmodi2021 need to be the reviewers of this...if they approve I'll merge
I ran into this earlier today and was initially confused b/c the docs mention that these fields should be optional but my stack trace showed otherwise. I'd also be happy if we can get this reviewed and merged 🙏
Also, can you rerun the test suite once rebased and tests fixed and share just the aggregated results part from the run result?? Thanks
Hey @anujmodi2021, thanks for the renewed interested in this PR. We have patched a fork on our side since this PR was created.
I have no idea how to run that test suite properly to be honest. I got inconsistent results last time I tried and was really confused as to how to run it correctly. If you have more experience with it, do you mind trying on your side ?
Hey @anujmodi2021, thanks for the renewed interested in this PR. We have patched a fork on our side since this PR was created.
I have no idea how to run that test suite properly to be honest. I got inconsistent results last time I tried and was really confused as to how to run it correctly. If you have more experience with it, do you mind trying on your side?
I can surely assist you on how to run the tests, but rebasing the PR to the latest trunk is something you only have to do. This PR anyway can't be merged without resolving conflicts...
Once you have rebased the PR, I will surely help you with the test suite run.