HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits

[shameersss1]

Description

The following are the operations which happens within a Task when it uses S3A Magic Committer.

During closing of stream

1. A 0-byte file with a same name of the original file is uploaded to S3 using PUT operation. Refer here for more information. This is done so that the downstream application like Spark could get the size of the file which is being written.

2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer here for more information.

During TaskCommit

1. All the MPU metadata which the task wrote to S3 (There will be 'x' number of metadata file in S3 if a single task writes to 'x' files) are read and rewritten to S3 as a single metadata file. Refer here for more information

Since these operations happens with the Task JVM, We could optimize as well as save cost by storing these information in memory when Task memory usage is not a constraint. Hence the proposal here is to introduce a new MagicCommit Tracker called "InMemoryMagicCommitTracker" which will store the

1. Metadata of MPU in memory till the Task is committed
2. Store the size of the file which can be used by the downstream application to get the file size before it is committed/visible to the output path.

This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call given a Task writes only 1 file.

Testing

1. Ran S3A integration test in us-west-1 region using the following command mvn -Dparallel-tests clean verify -Dit.test=ITestMagicCommitProtocol,ITestS3ACommitterMRJob,ITestMagicCommitProtocolFailure,ITestS3AHugeMagicCommits,ITestCommitOperationCost,ITestCommitOperations -Dtest=none -DtestsThreadCount=7

2. Manual verfication

3. Added Parameterized UnitTest in ITestMagicCommitProtocol

[shameersss1]

@steveloughran - I have converted draft PR to final one. Could you please review the changes. Thanks

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	48m 33s		trunk passed
+1 :green_heart:	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 30s		trunk passed
+1 :green_heart:	mvnsite	0m 39s		trunk passed
+1 :green_heart:	javadoc	0m 27s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 9s		trunk passed
+1 :green_heart:	shadedclient	37m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 29s		the patch passed
+1 :green_heart:	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 36s		the patch passed
+1 :green_heart:	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 26s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 12 new + 5 unchanged - 0 fixed = 17 total (was 5)
+1 :green_heart:	mvnsite	0m 30s		the patch passed
+1 :green_heart:	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
-1 :x:	javadoc	0m 24s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
-1 :x:	spotbugs	1m 10s	/new-spotbugs-hadoop-tools_hadoop-aws.html	hadoop-tools/hadoop-aws generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 :green_heart:	shadedclient	37m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	2m 54s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 33s		The patch does not generate ASF License warnings.
		140m 50s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-aws
	org.apache.hadoop.fs.s3a.commit.magic.InMemoryMagicCommitTracker.taskAttemptIdToBytesWritten isn't final but should be At InMemoryMagicCommitTracker.java:be At InMemoryMagicCommitTracker.java:[line 52]
	org.apache.hadoop.fs.s3a.commit.magic.InMemoryMagicCommitTracker.taskAttemptIdToMpuMetdadataMap isn't final but should be At InMemoryMagicCommitTracker.java:be At InMemoryMagicCommitTracker.java:[line 49]
	org.apache.hadoop.fs.s3a.commit.magic.InMemoryMagicCommitTracker.taskAttemptIdToPath isn't final but should be At InMemoryMagicCommitTracker.java:be At InMemoryMagicCommitTracker.java:[line 55]
Failed junit tests	hadoop.fs.s3a.commit.TestMagicCommitTrackerUtils

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 9ff01a4a6fb0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / ac490027753bcd60ae7de8d6a7dee30db4056501
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/2/testReport/
Max. process+thread count	528 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 1s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	45m 57s		trunk passed
+1 :green_heart:	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 31s		trunk passed
+1 :green_heart:	mvnsite	0m 40s		trunk passed
+1 :green_heart:	javadoc	0m 25s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 8s		trunk passed
+1 :green_heart:	shadedclient	37m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 29s		the patch passed
+1 :green_heart:	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 36s		the patch passed
+1 :green_heart:	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 26s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 20s		the patch passed
+1 :green_heart:	mvnsite	0m 33s		the patch passed
+1 :green_heart:	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
-1 :x:	javadoc	0m 24s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08 with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 :green_heart:	spotbugs	1m 11s		the patch passed
+1 :green_heart:	shadedclient	39m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	3m 4s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 34s		The patch does not generate ASF License warnings.
		140m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/3/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 3dde19b6ae38 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / a5b27c54f2bc3d75947fefcc5d040345231f91ce
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/3/testReport/
Max. process+thread count	616 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 1s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	46m 42s		trunk passed
+1 :green_heart:	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 31s		trunk passed
+1 :green_heart:	mvnsite	0m 41s		trunk passed
+1 :green_heart:	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 7s		trunk passed
+1 :green_heart:	shadedclient	37m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 29s		the patch passed
+1 :green_heart:	compile	0m 35s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 35s		the patch passed
+1 :green_heart:	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 26s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 20s		the patch passed
+1 :green_heart:	mvnsite	0m 31s		the patch passed
+1 :green_heart:	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 8s		the patch passed
+1 :green_heart:	shadedclient	37m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	2m 53s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 33s		The patch does not generate ASF License warnings.
		138m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/4/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 629db3c4380a 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 02fc9f187779a14e1645d7866c0f00423474c97c
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/4/testReport/
Max. process+thread count	527 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[shameersss1]

@steveloughran - Could you please review the changes?

[steveloughran]

I've thought about this some more. Here are some things which I believe we need

1. Marker files at the end of each path so that spark status reporting on different processes can get an update on an active job.
2. A way to abort all uploads of a failed task attempt -even from a different process. Probably also a way to abort the entire job.
3. Confidence that the inner memory store of pending uploads Will not grow it definitely.

Ignoring item number #3 for now, remember that we have #1 solved by adding a 0 byte marker with a header of "final length"; spark has some special handling zero byte files to use getXattr() and fall back to the probe for this -at the expense of a second HEAD request. Generating a modified FileStatus response from a single HEAD/getObjectMetadata() call Wood actually eliminate the need for that I wish I'd thought of it myself. Yes, we do break that guarantee that files listed are the same size as the files opened… but magic paths are, well, magic. We break a lot of guarantees there already.

The existing design should be retained even in memory; the calculation of final length something which can be done for all.

But: we do not need to save the .pending files just for task abort. All we need to do is be able to enumerate the upload IDs of all the files from that task attempt and cancel them. We can do that just by adding another header to the marker file. Task committee uses the memory data; task abort will need a deep scan of the task attempt, and all zero bite files with the proposed new header used to initiate water operations. This is only for task board an outlier case. For normal task commit there is no need to Scan the directory pause the pending files then generate a new pending set file for later pause commit. It is probably the Jason on the marshalling which is as much a performance killer here as the listing operation.

What do you think?

[shameersss1]

@steveloughran - Thanks a lot a for a detailed review and some amazing question, The following are my thoughts on the different asks.

1. Marker files at the end of each path so that spark status reporting on different processes can get an update on an active job.

As far i know (Please correct me if i am wrong)

1. 0-size marker files was specially added for Spark's use case.
2. After writing files, Spark tries to get the size of the files written for the statistic purpose (like showing the output bytes written) in the Spark History server UI.
3. This operation is being done as part of the BasicWriteStatsTracker class in Spark.
4. As i could see in my experiment, BasicWriteStatsTracker#getFileSize is called in the executor process itself.

That being said, Since the same process is calling BasicWriteStatsTracker#getFileSize is it still required to have 0 marker file? I have solved this by adding a check in FileStatus method by returing the file size corresponding to the magic path/file.

---

2. A way to abort all uploads of a failed task attempt -even from a different process. Probably also a way to abort the entire job.

Thinking from Spark's perspective,

1. When a taskAttempt fails (gracefully), abortTask operation is called. This is operation is called within the same process and hence we can fetch the MPU metadata from the memory itself.

2. If a taskAttempt fails (ungracefully and all retries) are exhausted, When abortJob operation is called which will internally invoke cleanup which lists all the pending multi part upload and aborts them.

That being said, I am not sure if there is any such use case of abortingTask from another process. In such cases, The abortJob will handle it i guess.

---

3. Confidence that the inner memory store of pending uploads Will not grow it definitely.

1. The static map entry is removed is during taskCommit or abortTask operations and hence it guaranteed that there is no memory leak (unless there is some unexplored corner case).

2. The only case when it grows large, is when there are large number of concurrent jobs reusing the same executor JVM, Since we don't enable the "inmemory" by default we should be good. That being said, maybe we should call this out in the documentation.

---

Doe it make sense? Or am i missing anything?

[shameersss1]

1. static map of path to metadata. This will grow without constraint on a long live process.

The entries to the Map are removed during commitTask or abortTask operation to keep memory under control.

---

2. Two jobs writing to same path will it corrupt the Map ?

No, The path (complete) is guaranteed to be unique The paths stored here as part of private static Map<String, List<Path>> taskAttemptIdToPath = new ConcurrentHashMap<>(); is the magic path, Eventhough the file name might be same, The magic path for two different jobs will be different since the jobId is included in the path.

---

3. the static map would be a weak ref to something held strongly by the actual committer (see WeakReferenceMap). Once the actual task attempt is gc'd,

Since the entries from the HashMap are removed during commitTask or abortTask operation is WeakHashMap still required?

---

4. static structures should be per fs instances, so when an fs is cleaned up

I am not sure why it should be scoped under fs object. For a simiar behaviour with storing in s3, Shouldn't the static structure be available to the whole JVM ? I mean shouldn't we able to access static structure irrespective of the fs object.

---

5. 'm also worried about how a job could abort a task attempt on a different process which has failed. Before worrying about that too much, why don't you look in spark to see how it calls abort. I'm not worried about MapReduce except for testing -so how do itself calls the committee isn't so important. For example: we don't care about recovery from a failed attempt as spark itself cannot do this.

I have covered this as part of the comment here.

---

[shameersss1]

@steveloughran - Thanks a lot a detailed review as well as amazing follow up question. I have addressed your comments, Please let me know your thoughts.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	46m 54s		trunk passed
+1 :green_heart:	compile	0m 44s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 32s		trunk passed
+1 :green_heart:	mvnsite	0m 43s		trunk passed
+1 :green_heart:	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 9s		trunk passed
+1 :green_heart:	shadedclient	39m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 32s		the patch passed
+1 :green_heart:	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 38s		the patch passed
+1 :green_heart:	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 27s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	0m 22s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 :green_heart:	mvnsite	0m 37s		the patch passed
+1 :green_heart:	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 12s		the patch passed
+1 :green_heart:	shadedclient	38m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	2m 55s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 35s		The patch does not generate ASF License warnings.
		141m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/5/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux f10aab5720a0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 7d213000e14bcd1ee7aa6c1369f996a0815347e8
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/5/testReport/
Max. process+thread count	528 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[shameersss1]

@steveloughran - Gentle reminder for review

[shameersss1]

@steveloughran - Thanks a lot for review comments, I have addressed the comments with the new commit d8db729e5568df5dc920604eff8167a575a5894c

1. Added test in ITestTerasortOnS3A
2. Fixed import ordering
3. Addressed other minor comments

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	46m 39s		trunk passed
+1 :green_heart:	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 30s		trunk passed
+1 :green_heart:	mvnsite	0m 40s		trunk passed
+1 :green_heart:	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 9s		trunk passed
+1 :green_heart:	shadedclient	37m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 28s		the patch passed
+1 :green_heart:	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 36s		the patch passed
+1 :green_heart:	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 25s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 20s		the patch passed
+1 :green_heart:	mvnsite	0m 30s		the patch passed
+1 :green_heart:	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 5s		the patch passed
+1 :green_heart:	shadedclient	37m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	2m 48s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 33s		The patch does not generate ASF License warnings.
		138m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/6/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 29f0f609a7d0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d8db729e5568df5dc920604eff8167a575a5894c
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/6/testReport/
Max. process+thread count	526 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[shameersss1]

@steveloughran - Gentle reminder to review the changes. Thanks

[steveloughran]

@shameersss1 I'm working on assisting getting the 3.4.0 release out right now. Anything you can do to assist testing would be wonderful, as I'm only worrying about release blockers.

[shameersss1]

@steveloughran - I am glad to assist with the testing. Is there any release candidate branch for the same? Could you please share the wiki on what tests needs to be done?

[shameersss1]

@steveloughran - Gentle reminder for review Thanks

[shameersss1]

@steveloughran - Thanks a lot for the detailed review. I have addressed your comments.

note, you will need to a followup in the docs -but we can get this in and tested while you do that... What docs are we mentioning here? I have added the details in the committer.md file though.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	49m 35s		trunk passed
+1 :green_heart:	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart:	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart:	checkstyle	0m 33s		trunk passed
+1 :green_heart:	mvnsite	0m 42s		trunk passed
+1 :green_heart:	javadoc	0m 27s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart:	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart:	spotbugs	1m 8s		trunk passed
+1 :green_heart:	shadedclient	38m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 :x:	mvninstall	0m 15s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 :x:	compile	0m 16s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	javac	0m 16s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	compile	0m 16s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
-1 :x:	javac	0m 16s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	0m 15s	/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 :x:	mvnsite	0m 17s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 :green_heart:	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
-1 :x:	javadoc	0m 17s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
-1 :x:	spotbugs	0m 16s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 :green_heart:	shadedclient	40m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	0m 20s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 :green_heart:	asflicense	0m 36s		The patch does not generate ASF License warnings.
		139m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/7/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 68ca386861b1 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 7239b2f75d705ea79c8ce7450b5fe4dec18a1f33
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/7/testReport/
Max. process+thread count	603 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[shameersss1]

@steveloughran - I have addressed your comments.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	49m 2s		trunk passed
+1 :green_heart:	compile	0m 45s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart:	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart:	checkstyle	0m 32s		trunk passed
+1 :green_heart:	mvnsite	0m 42s		trunk passed
+1 :green_heart:	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart:	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart:	spotbugs	1m 8s		trunk passed
+1 :green_heart:	shadedclient	38m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 30s		the patch passed
+1 :green_heart:	compile	0m 37s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart:	javac	0m 37s		the patch passed
+1 :green_heart:	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart:	javac	0m 27s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13)
+1 :green_heart:	mvnsite	0m 32s		the patch passed
+1 :green_heart:	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart:	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart:	spotbugs	1m 9s		the patch passed
+1 :green_heart:	shadedclient	38m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	2m 49s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 38s		The patch does not generate ASF License warnings.
		144m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/8/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6468
JIRA Issue	HADOOP-19047
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 6a4507cb7348 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 8d739beced49d1469593671cd7412b4662ca2f04
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/8/testReport/
Max. process+thread count	527 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6468/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.