hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-14837 : Support Read Restored Glacier Objects
Description of PR
Currently S3A does not distinguish Glacier and Glacier Deep Archive files, as it doesn't examine the storage class or verify if an object is in the process of restoration from Glacier. Attempting to access an in-progress Glacier file via S3A results in an AmazonS3Exception, indicating the operation's invalidity for the object's storage class.
As part of this change, Users will be able to successfully read restored glacier objects from the s3 location of the table using S3A. It will ignore any Glacier archived files if they are in process of being restored asynchronously. There will be no change to the existing behavior and additional configuration will be needed to enable the above mentioned flow.
The config which would control the behavior of the S3AFileSystem with respect to glacier storage classes will be fs.s3a.glacier.read-restored-objects
The config can have 3 values:
-
READ_ALL: This would conform to the current default behavior of not taking into account the storage classes retrieved from S3. This will be done to keep the current behavior for the users and not changing the experience for them. -
SKIP_ALL_GLACIER: If this value is set then this will ignore any S3 Objects which are tagged with Glacier storage classes and retrieve the others. -
READ_RESTORED_GLACIER_OBJECTS: If this value is set then restored status of the Glacier object will be checked, if restored the objects would be read like normal S3 objects else they will be ignored as the objects would not have been retrieved from the S3 Glacier. ( To check the restored status, newly introduced RestoredStatus will be used which is present in the S3 Object). This wasn't previously possible as ListObjects did not return any information about the restore status of an object, only it's storage class.
A new FileStatusAcceptor is created which will use the RestoreStatus attribute from the S3Object and will filter out or include the glacier objects from the list as defined by the config. FileStatusAcceptor is an interface with 3 overloaded predicates, which filter the files based on the conditions defined in the said predicates. A new attribute RestoreStatus of will be used from the response of ListObjects . This field will indicate whether the object is unrestored, restoring, or restored, and also when the expiration of that restore is.
How was this patch tested?
Integration Tests (hadoop-aws)
All the Integration tests are passing, the tests were run in accordance with https://hadoop.apache.org/docs/current2/hadoop-aws/tools/hadoop-aws/testing.html. The tests were executed in the region us-east-1.
There were 2 failures observed which seem intermittent and unrelated to the change introduced in this CR. As the default behavior of S3AFileSystem was not changed.
Failures observed :
ITestS3ACommitterFactory.testEverything
ITestS3AConfiguration.testRequestTimeout
Manual Testing
Manual testing of the change was done with Spark v3.5
A Parquet table was created using the following in Spark-SQL.
CREATE DATABASE IF NOT EXISTS glacier_test location "s3a://<bucket>/data/glacier_test";
USE glacier_test;
CREATE TABLE IF NOT EXISTS parquet_glacier_test (id int, data string) using parquet location "s3a://<bucket>/data/glacier_test/parquet_glacier_test";
INSERT INTO parquet_glacier_test VALUES (1, 'a'), (2, 'b'), (3, 'c');
INSERT INTO parquet_glacier_test VALUES (4, 'a'), (5, 'b'), (6, 'c');
INSERT INTO parquet_glacier_test VALUES (7, 'a'), (8, 'b'), (9, 'c');
Was able to successfully retrieve the data using the following.
SELECT * FROM parquet_glacier_test;
+---+----+
| id|data|
+---+----+
| 7| a|
| 8| b|
| 9| c|
| 4| a|
| 5| b|
| 6| c|
| 1| a|
| 2| b|
| 3| c|
+---+----+
The storage class of the file s3://<bucket>/data/glacier_test/parquet_glacier_test/part-00000-f9cb400e-35b2-41f7-9c39-8e34cd830fed-c000.snappy.parquet was changed to Glacier Flexible Retrieval (formerly Glacier) from Standard.
When trying to retrieve the data again form the same table, the following exception was observed.
software.amazon.awssdk.services.s3.model.InvalidObjectStateException: The operation is not valid for the object's storage class (Service: S3, Status Code: 403, Request ID: X05JDR633AAK4TBQ, Extended Request ID: uOxWdN4giUAuB9a4YWvnyrXPYCi2U35P5BrHhFO3aLSLLe4GtWhXGXCEJ/Ld5EyGr5b6VezTzeI=):InvalidObjectState
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:243)
at org.apache.hadoop.fs.s3a.Invoker.onceTrackingDuration(Invoker.java:149)
at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:278)
at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:425)
at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:284)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:408)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:404)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:282)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:326)
at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:417)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:536)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
I restarted the spark-sql session by setting the following config.
spark-sql --conf spark.hadoop.fs.s3a.glacier.read-restored-objects=SKIP_ALL_GLACIER
Trying to access the table now , resulted in the following.
SELECT * FROM parquet_glacier_test;
+---+----+
| id|data|
+---+----+
| 7| a|
| 8| b|
| 9| c|
| 4| a|
| 5| b|
| 6| c|
+---+----+
The spark-sql session was restarted with the following config
spark-sql --conf spark.hadoop.fs.s3a.glacier.read-restored-objects=READ_RESTORED_GLACIER_OBJECTS
Trying to access the table now , resulted in the same result as the previous step as unrestored glacier file was ignored when the table was read.
SELECT * FROM parquet_glacier_test;
+---+----+
| id|data|
+---+----+
| 7| a|
| 8| b|
| 9| c|
| 4| a|
| 5| b|
| 6| c|
+---+----+
The restore for the file s3://<bucket>/data/glacier_test/parquet_glacier_test/part-00000-f9cb400e-35b2-41f7-9c39-8e34cd830fed-c000.snappy.parquet was initiated using the S3 Console.
Trying to access the table now , resulted in the same result as the previous step as the glacier file was still being restored and was not available.
SELECT * FROM parquet_glacier_test;
+---+----+
| id|data|
+---+----+
| 7| a|
| 8| b|
| 9| c|
| 4| a|
| 5| b|
| 6| c|
+---+----+
On retrying after 5-7 minutes ( as it was expedited retrieval ) the following was the result, which is as expected :
SELECT * FROM parquet_glacier_test;
+---+----+
| id|data|
+---+----+
| 7| a|
| 8| b|
| 9| c|
| 4| a|
| 5| b|
| 6| c|
| 1| a|
| 2| b|
| 3| c|
+---+----+
For code changes:
- [Yes] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [Yes] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [NA] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
most of my comments are on the basic stuff, especially those test assertions and the need to have a single factored out assertion() method.
Am converting the current test into an ITest, will update assertions according to the recommendations
Now, can we have the storage class an attribute in S3AFileStatus? populated in listings and from HEAD requests? included in .toString()? that could be useful in future
This is something that can be done, do we want this as part of this PR ? Or a separate one adding the storage class to S3AFileStatus ?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 13m 44s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 22m 14s | trunk passed | |
| +1 :green_heart: | compile | 9m 50s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 8m 59s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 2m 29s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 20s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 59s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 46s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 2m 0s | trunk passed | |
| +1 :green_heart: | shadedclient | 21m 15s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 0m 15s | /patch-mvninstall-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | compile | 8m 45s | /patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 8m 45s | /patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 8m 9s | /patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javac | 8m 9s | /patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 2m 9s | /results-checkstyle-root.txt | root: The patch generated 22 new + 6 unchanged - 0 fixed = 28 total (was 6) |
| -1 :x: | mvnsite | 0m 24s | /patch-mvnsite-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| -1 :x: | spotbugs | 0m 25s | /patch-spotbugs-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 21m 14s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 16m 27s | hadoop-common in the patch passed. | |
| -1 :x: | unit | 0m 28s | /patch-unit-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | |
| 150m 53s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint |
| uname | Linux e8c67875814a 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4a9f038a786838dda1ff013ffd3e34a0d8e54ba1 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/3/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 22s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 13m 42s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 22m 18s | trunk passed | |
| +1 :green_heart: | compile | 9m 43s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 9m 4s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 2m 27s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 18s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 57s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 48s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 2m 3s | trunk passed | |
| +1 :green_heart: | shadedclient | 21m 11s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 0m 16s | /patch-mvninstall-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | compile | 8m 39s | /patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 8m 39s | /patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 8m 7s | /patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javac | 8m 7s | /patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 2m 12s | /results-checkstyle-root.txt | root: The patch generated 22 new + 6 unchanged - 0 fixed = 28 total (was 6) |
| -1 :x: | mvnsite | 0m 23s | /patch-mvnsite-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| -1 :x: | spotbugs | 0m 24s | /patch-spotbugs-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 21m 29s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 16m 28s | hadoop-common in the patch passed. | |
| -1 :x: | unit | 0m 28s | /patch-unit-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | |
| 150m 41s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint |
| uname | Linux 5fa6c0e6973e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4a9f038a786838dda1ff013ffd3e34a0d8e54ba1 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/4/testReport/ |
| Max. process+thread count | 1281 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 21s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 31m 57s | trunk passed | |
| +1 :green_heart: | compile | 0m 26s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 21s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 28s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 45s | trunk passed | |
| +1 :green_heart: | shadedclient | 19m 28s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 19s | the patch passed | |
| +1 :green_heart: | compile | 0m 20s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 20s | the patch passed | |
| +1 :green_heart: | compile | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 16s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 12s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 18 new + 6 unchanged - 0 fixed = 24 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 18s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 10s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 43s | the patch passed | |
| +1 :green_heart: | shadedclient | 19m 22s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 26s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | |
| 81m 44s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 4dfd5fc0d4f0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f0144a759ad6604af682fd3c28c60d303e7036d5 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/5/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 23s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 32m 31s | trunk passed | |
| +1 :green_heart: | compile | 0m 25s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 17s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 26s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 18s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 44s | trunk passed | |
| +1 :green_heart: | shadedclient | 19m 20s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 20s | the patch passed | |
| +1 :green_heart: | compile | 0m 21s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 21s | the patch passed | |
| +1 :green_heart: | compile | 0m 15s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 15s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 10s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 13 new + 6 unchanged - 0 fixed = 19 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 17s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 45s | the patch passed | |
| +1 :green_heart: | shadedclient | 19m 17s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 28s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | |
| 82m 58s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 732160df4277 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4e9b673f0ace4ebaa8c1e467617b6fe5d8ab93b4 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/6/testReport/ |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | patch | 0m 16s | https://github.com/apache/hadoop/pull/6407 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/7/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 21s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 31m 31s | trunk passed | |
| +1 :green_heart: | compile | 0m 23s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 19s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 19s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 23s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 17s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 41s | trunk passed | |
| +1 :green_heart: | shadedclient | 19m 21s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 16s | the patch passed | |
| +1 :green_heart: | compile | 0m 20s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 20s | the patch passed | |
| +1 :green_heart: | compile | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 16s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 13s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 21 new + 6 unchanged - 0 fixed = 27 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 18s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 10s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 41s | the patch passed | |
| +1 :green_heart: | shadedclient | 19m 5s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 27s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | |
| 81m 4s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 6f1c70e86dea 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d035d1561d64e8d1279a7c6240bff6f09494b64c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/8/testReport/ |
| Max. process+thread count | 663 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
I 'd go for getTrimmed(READ_RESTORED_GLACIER_OBJECTS, ""); if empty string map to empty optional, otherwise .toupper and valueof. one thing to consider: meaningful failure if the value doesn't map.
or we just go for "upper case is required" and use what you've proposed. more brittle but simpler?
Regarding the above, I will update the config read to getTrimmed and set the config values of defualt to READ_ALL instead of "".
meaningful failure if the value doesn't map
the valueof method will throw the illegalArgs exception if an invalid value is set in the config
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 23s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| -1 :x: | mvninstall | 1m 40s | /branch-mvninstall-root.txt | root in trunk failed. |
| -1 :x: | compile | 0m 22s | /branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | hadoop-aws in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 3m 21s | /branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | hadoop-aws in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -0 :warning: | checkstyle | 0m 19s | /buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 21s | /branch-mvnsite-hadoop-tools_hadoop-aws.txt | hadoop-aws in trunk failed. |
| -1 :x: | javadoc | 0m 22s | /branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | hadoop-aws in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 5s | /branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | hadoop-aws in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | spotbugs | 1m 45s | trunk passed | |
| -1 :x: | shadedclient | 4m 27s | branch has errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| -1 :x: | mvninstall | 0m 20s | /patch-mvninstall-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 20s | /patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | hadoop-aws in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 20s | /patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt | hadoop-aws in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 30s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 20s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 27 new + 0 unchanged - 0 fixed = 27 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 19s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 11s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 14s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 39s | the patch passed | |
| -1 :x: | shadedclient | 1m 11s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 0m 22s | /patch-unit-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +0 :ok: | asflicense | 0m 20s | ASF License check generated no output? | |
| 19m 52s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux b4c0370956e2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e110e2ab0a2b50b58efb66d509efedce1b376f7b |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/9/testReport/ |
| Max. process+thread count | 88 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Build failed due to an OOM Error,
build passing on local
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:33 min
[INFO] Finished at: 2024-02-07T08:13:12Z
[INFO] ------------------------------------------------------------------------
[ERROR] unable to create new native thread -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 20s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 31m 45s | trunk passed | |
| +1 :green_heart: | compile | 0m 25s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 20s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 25s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 18s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 43s | trunk passed | |
| +1 :green_heart: | shadedclient | 19m 36s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 17s | the patch passed | |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 19s | the patch passed | |
| +1 :green_heart: | compile | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 16s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 12s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 21 new + 6 unchanged - 0 fixed = 27 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 18s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 42s | the patch passed | |
| +1 :green_heart: | shadedclient | 19m 1s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 28s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | |
| 81m 32s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 8e4ff0d8df99 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c5dcd5c4426135b7b054ed19f4bcf8e68d955d77 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/10/testReport/ |
| Max. process+thread count | 559 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
checkstyle failures, we all get those. and sometimes we have to say "better if broken"
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/api/RequestFactory.java:256: /**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java:617: .optionalObjectAttributes(OptionalObjectAttributes.RESTORE_STATUS) // Optional Attribute to get the Restored Status of the Glacier Objects: Line is longer than 100 characters (found 146). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StoreContext.java:419: /**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StoreContext.java:420: * Return the S3ObjectStorageClassFilter object for S3A, whose value is set according to the config {@code fs.s3a.glacier.read.restored.objects}: Line is longer than 100 characters (found 146). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:467: // Handle Glacier Storage Class objects based on the config fs.s3a.glacier.read.restored.objects value set: Line is longer than 100 characters (found 114). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:468: if ( s3ObjectStorageClassFilter.getFilter().apply(s3Object) && acceptor.accept(keyPath, s3Object) && filter.accept(keyPath)) {: Line is longer than 100 characters (found 135). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:468: if ( s3ObjectStorageClassFilter.getFilter().apply(s3Object) && acceptor.accept(keyPath, s3Object) && filter.accept(keyPath)) {:12: '(' is followed by whitespace. [ParenPad]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:30: * {@link S3ObjectStorageClassFilter} will filter the S3 files based on the {@code fs.s3a.glacier.read.restored.objects} configuration set in {@link S3AFileSystem}: Line is longer than 100 characters (found 163). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:32: * {@code READ_ALL}: Retrieval of Glacier files will fail with InvalidObjectStateException: The operation is not valid for the object's storage class.: Line is longer than 100 characters (found 150). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:33: * {@code SKIP_ALL_GLACIER}: If this value is set then this will ignore any S3 Objects which are tagged with Glacier storage classes and retrieve the others.: Line is longer than 100 characters (found 157). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:34: * {@code READ_RESTORED_GLACIER_OBJECTS}: If this value is set then restored status of the Glacier object will be checked, if restored the objects would be read like normal S3 objects else they will be ignored as the objects would not have been retrieved from the S3 Glacier.: Line is longer than 100 characters (found 275). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:42: private static final Set<ObjectStorageClass> GLACIER_STORAGE_CLASSES = Sets.newHashSet(ObjectStorageClass.GLACIER, ObjectStorageClass.DEEP_ARCHIVE);: Line is longer than 100 characters (found 150). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:50: /**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:59: /**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ObjectStorageClassFilter.java:68: /**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/list/ITestS3AReadRestoredGlacierObjects.java:70: private final int MAX_RETRIES = 100;:21: Name 'MAX_RETRIES' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MemberName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/list/ITestS3AReadRestoredGlacierObjects.java:71: private final int RETRY_DELAY_MS = 5000;:21: Name 'RETRY_DELAY_MS' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MemberName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/list/ITestS3AReadRestoredGlacierObjects.java:84: conf.set(STORAGE_CLASS, glacierClass); // Create Glacier objects:Storage Class:DEEP_ARCHIVE/GLACIER: Line is longer than 100 characters (found 103). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/list/ITestS3AReadRestoredGlacierObjects.java:118: try (FileSystem fs = createFiles(S3ObjectStorageClassFilter.READ_RESTORED_GLACIER_OBJECTS.name())) {: Line is longer than 100 characters (found 104). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/list/ITestS3AReadRestoredGlacierObjects.java:128: Assume.assumeTrue(type == Type.GLACIER); // Skipping this test for Deep Archive as expedited retrieval is not supported: Line is longer than 100 characters (found 123). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/list/ITestS3AReadRestoredGlacierObjects.java:129: try (FileSystem fs = createFiles(S3ObjectStorageClassFilter.READ_RESTORED_GLACIER_OBJECTS.name())) {: Line is longer than 100 characters (found 104). [LineLength]
Proposed not fixing
- ITestS3AReadRestoredGlacierObjects.java:129
- ITestS3AReadRestoredGlacierObjects.java:118
others should be addressed. you can save time by doing this locally
mvn checkstyle:check
just look at the files you've edited and don't worry about the rest.
@bpahuja can you not do rebase and force push unless some other change has broken the merge? breaks the "changes since your last review" feature. If someone has stepped on the code, don't be afraid to rebase -I generally squash all changes first there to make the rebase easier
Hi @steveloughran, Have addressed the checkstyle violations in the latest commit.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 20s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 31m 18s | trunk passed | |
| +1 :green_heart: | compile | 0m 25s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 18s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 26s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 42s | trunk passed | |
| +1 :green_heart: | shadedclient | 18m 51s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 17s | the patch passed | |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 19s | the patch passed | |
| +1 :green_heart: | compile | 0m 15s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 15s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 10s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 2 new + 6 unchanged - 0 fixed = 8 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 18s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 40s | the patch passed | |
| +1 :green_heart: | shadedclient | 18m 49s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 26s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | |
| 79m 33s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux cc61e9a0a8ab 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2e6562a3ba9691060f7b76fd51dcfa554469652c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/11/testReport/ |
| Max. process+thread count | 626 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 7m 18s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 31m 16s | trunk passed | |
| +1 :green_heart: | compile | 0m 23s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 17s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 19s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 26s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 17s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 41s | trunk passed | |
| +1 :green_heart: | shadedclient | 18m 49s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 19m 2s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 18s | the patch passed | |
| +1 :green_heart: | compile | 0m 20s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 20s | the patch passed | |
| +1 :green_heart: | compile | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 16s | the patch passed | |
| -1 :x: | blanks | 0m 0s | /blanks-eol.txt | The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 10s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 3 new + 6 unchanged - 0 fixed = 9 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 18s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| -1 :x: | javadoc | 0m 16s | /patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt | hadoop-aws in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | spotbugs | 0m 39s | the patch passed | |
| +1 :green_heart: | shadedclient | 18m 50s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 22s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | |
| 86m 21s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 92c6e03fa6e0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 566292b7005113772344d20173815ba4805c774a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/12/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 21s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 33m 55s | trunk passed | |
| +1 :green_heart: | compile | 0m 25s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 18s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 25s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 18s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 44s | trunk passed | |
| +1 :green_heart: | shadedclient | 19m 52s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 20m 5s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 18s | the patch passed | |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 19s | the patch passed | |
| +1 :green_heart: | compile | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 16s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 13s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 3 new + 6 unchanged - 0 fixed = 9 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 19s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 39s | the patch passed | |
| +1 :green_heart: | shadedclient | 19m 41s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 14s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | |
| 84m 10s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux e4e0405a30ee 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6742e8d0e79057623ea74c1d040be93653f3f2cd |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/13/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@bpahuja have you tested this with any of: s3 express, third party stores, gcs? Just want to make sure things work and the tests are skipped?
I'll inevitably do the test runs anyway, but it'd save me time and effort, and as I don't run those regularly enough, multiple commits may be the source of regressions
@bpahuja have you tested this with any of: s3 express, third party stores, gcs? Just want to make sure things work and the tests are skipped?
Hello @steveloughran , wanted to confirm the requirement here ? Is it that you wanted me to ensure that this test is skipped when ran in other environments ?
If that's the case then IMO, it should work as other Integration tests do in this package as it is being inheriting AbstractS3ATestBase.
Do let me know if something other then this is needed from me :)
Thanks
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 23s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 32m 13s | trunk passed | |
| +1 :green_heart: | compile | 0m 23s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 19s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 23s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 17s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 44s | trunk passed | |
| +1 :green_heart: | shadedclient | 19m 11s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 19m 23s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 19s | the patch passed | |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 19s | the patch passed | |
| +1 :green_heart: | compile | 0m 14s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 14s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 11s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 2 new + 6 unchanged - 0 fixed = 8 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 19s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 40s | the patch passed | |
| +1 :green_heart: | shadedclient | 18m 42s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 13s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | |
| 80m 59s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 7c86bd58d065 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df48a67dcfcd9ef4578c2133660ced4924986e92 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/14/testReport/ |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 36m 54s | trunk passed | |
| +1 :green_heart: | compile | 0m 28s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 23s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 48s | trunk passed | |
| +1 :green_heart: | shadedclient | 22m 54s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 23m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 20s | the patch passed | |
| +1 :green_heart: | compile | 0m 20s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 20s | the patch passed | |
| +1 :green_heart: | compile | 0m 15s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 15s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 14s | /results-checkstyle-hadoop-tools_hadoop-aws.txt | hadoop-tools/hadoop-aws: The patch generated 2 new + 6 unchanged - 0 fixed = 8 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 19s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 11s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 0m 47s | the patch passed | |
| +1 :green_heart: | shadedclient | 22m 6s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 15s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | |
| 93m 44s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux fe484a3a2b53 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cbb8580f8732c7bd6cf5902b024d5545b4950295 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/15/testReport/ |
| Max. process+thread count | 556 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
I'm doing nothing but helping it get Hadoop 3.4.0 out the door this week. No code of my except related to packaging; no reviews of other people except related to the release at all. Sorry.
Well you are waiting for a review on your code from myself or someone else -why not getting involved? There is no release engineering team doing this and it is up to all of us developers in the community to get our hands dirty. We all have different deployment environments and we all have different things we want to test. And, given this is the first public release with the AWS V2 SDK: it matters a lot that this thing ships. It will be the way we actually find real world bugs.
Look on the hadoop common-dev list for the announcement of the next release candidate: once the announcement is made we have five days to test and vote on the RC. That is why we are under so much time pressure here.
Note I've created a project, https://github.com/apache/hadoop-release-support , to assist in qualifying. One thing which would be good it would be some extra scripts we could run to help validate storage operations -operations which we can then execute against cloud storage from either the local host or a remote one -I am qualifying the Arm64 build on a raspberry pi5 under my television and would like to have that testing fully automated.
Anything you can do here will be very much appreciated. And like I said: I'm unlikely to be looking at any other code right now.
Hello @steveloughran, Just a gentle reminder to review the PR :) , Do take a look whenever you get some time
Thanks
I am catching up now, as warned giving priority to people who helped validate the RC. the core project is a community project, and everyone gets to participate
@bpahuja can you address the review comments? Otherwise I'm going to forget about it.
Get it done now and we can target 3.4.1
@steveloughran , I have pushed the changes for ITestS3AReadRestoredGlacierObjects.java , and it now has only 1 test with all the test cases. In total , the setup happens twice, once for Glacier Storage class and once more for Glacier Deep Archive Storage class. The number of setups happening has been reduced , where earlier it created the file for each test case.
Please review whenever you get some time.
Let me know if any more changes are needed here.
Thanks :)
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 23s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 36m 30s | trunk passed | |
| +1 :green_heart: | compile | 0m 28s | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | |
| +1 :green_heart: | compile | 0m 24s | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 25s | trunk passed | |
| -1 :x: | javadoc | 0m 24s | /branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt | hadoop-aws in trunk failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 27s | /branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt | hadoop-aws in trunk failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 0m 26s | /branch-spotbugs-hadoop-tools_hadoop-aws.txt | hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 3m 0s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 3m 24s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| -1 :x: | mvninstall | 0m 21s | /patch-mvninstall-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 22s | /patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt | hadoop-aws in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 0m 22s | /patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt | hadoop-aws in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 22s | /patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt | hadoop-aws in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 22s | /patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt | hadoop-aws in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 20s | /buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 22s | /patch-mvnsite-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 0m 5s | /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt | hadoop-aws in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 20s | /patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt | hadoop-aws in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 0m 22s | /patch-spotbugs-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 3m 17s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 0m 22s | /patch-unit-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? | |
| 50m 21s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 2f4a590eb200 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c1c72d9ce06b06f60fd73d62e5afe8b650b1a602 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/16/testReport/ |
| Max. process+thread count | 77 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6407/16/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.