HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags

[steveloughran]

HADOOP-19161

Initial design

• no tests or docs
• served up via StoreContext. Not sure about the merits of that I think it is needed so it gets down to all AbstractStoreOperation instances, but should that be where the decision is made?
• create performance is wired up.
• as is path capabilities

For testing we need to make sure ths is unset from all cost tests.

relates to #6543; the logic to set up that operation is here...that PR would just be the implementation.

Same for a delete optimisation where we'd skip parent dir probe. rename could do the same for its source dir too.

For code changes:

• [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[steveloughran]

note the commented out bit where we considered adding options like "hive" or "spark".

@HarshitGupta11 and I discussed this; for now lets go with a list of options and "*"

[steveloughran]

I have a better design for this. changign this to draft.

Proposed: we have a Configuration.getEnumOptions(Enum x, boolean failIfUnknown) which returns an EnumSet of all values of the enum class whose valueOf() matches an entry in the CSV list (with some mapping such as case conversion, and map - and . to "_".

this makes it trivial to reuse/process. the implementation would be outside the actual Configuration class to make it easy for AbfsConfiguration to use too

[steveloughran]

Reason: Use hadoop-common provided Sets rather than Guava provided Sets
	in file: org/apache/hadoop/util/ConfigurationUtil.java
		org.apache.hadoop.thirdparty.com.google.common.collect.Sets 	(Line: 33, Matched by: org.apache.hadoop.thirdparty.com.google.common.collect.Sets)

[steveloughran]

@mukund-thakur @saxenapranav @HarshitGupta11 @ahmarsuhail @virajjasani

FYI, this is ready to review. Once in, viraj's mkdirs option can go in, and then we can document this option in performance.md

ABFS can use it for their options

[steveloughran]

OK, I'm happy with this. @saxenapranav can you look at this? this flagset is what I hope to use for a similar set of options in abfs -which is why the parser lives outside of Configuration.

@mukund-thakur @virajjasani thoughts?

[virajjasani]

Sounds good, i would take a detailed look at ConfigurationHelper and FlagSet soon.

[steveloughran]

realised we need one more thing: a way to enumerate all enabled flags. This is to make the bucketInfo list of flags easy to self-generate

[steveloughran]

@virajjasani will adopt your comments next week, will need to rebase against the changes done by one "stevel" who appears to have broken this merging

[steveloughran]

@virajjasani done your comments; had to rebase for what turned out to be a spurious merge conflict (adjacent methods in s3afs), so you can't see exactly what changed in the final commit. suffice to say: did your requests except related to builder and mutability.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 1s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 :ok:	mvndep	14m 44s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	32m 21s		trunk passed
+1 :green_heart:	compile	17m 21s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	16m 7s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	4m 30s		trunk passed
+1 :green_heart:	mvnsite	2m 41s		trunk passed
+1 :green_heart:	javadoc	1m 57s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 44s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 57s		trunk passed
+1 :green_heart:	shadedclient	34m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 33s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	1m 28s		the patch passed
+1 :green_heart:	compile	16m 42s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	16m 42s		the patch passed
+1 :green_heart:	compile	15m 54s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	15m 54s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	4m 25s	/results-checkstyle-root.txt	root: The patch generated 2 new + 116 unchanged - 0 fixed = 118 total (was 116)
+1 :green_heart:	mvnsite	2m 40s		the patch passed
+1 :green_heart:	javadoc	1m 51s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 45s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	4m 20s		the patch passed
+1 :green_heart:	shadedclient	34m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	19m 48s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 53s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	1m 6s		The patch does not generate ASF License warnings.
		242m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6789
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 937d04fdd13c 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 4b5f98533dc341d594402862bf648da25398b2ff
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/testReport/
Max. process+thread count	3152 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	17m 39s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 :ok:	mvndep	14m 19s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	37m 35s		trunk passed
+1 :green_heart:	compile	18m 42s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	17m 47s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	4m 36s		trunk passed
+1 :green_heart:	mvnsite	2m 33s		trunk passed
+1 :green_heart:	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 45s		trunk passed
+1 :green_heart:	shadedclient	39m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 31s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	1m 25s		the patch passed
+1 :green_heart:	compile	18m 50s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	18m 50s		the patch passed
+1 :green_heart:	compile	18m 10s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	18m 10s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	5m 22s	/results-checkstyle-root.txt	root: The patch generated 2 new + 116 unchanged - 0 fixed = 118 total (was 116)
+1 :green_heart:	mvnsite	2m 43s		the patch passed
+1 :green_heart:	javadoc	1m 45s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 43s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	4m 43s		the patch passed
+1 :green_heart:	shadedclient	43m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	20m 4s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 43s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 59s		The patch does not generate ASF License warnings.
		285m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6789
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 8c687f3b19c4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 4b5f98533dc341d594402862bf648da25398b2ff
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/testReport/
Max. process+thread count	3137 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:997:        .setPerformanceFlags(FlagSet.createFlagSet(PerformanceFlagEnum.class,FS_S3A_PERFORMANCE_FLAGS)): Line is longer than 100 characters (found 103). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:997:        .setPerformanceFlags(FlagSet.createFlagSet(PerformanceFlagEnum.class,FS_S3A_PERFORMANCE_FLAGS)):77: ',' is not followed by whitespace. [WhitespaceAfter]

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 1s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 :ok:	mvndep	14m 57s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	32m 24s		trunk passed
+1 :green_heart:	compile	17m 24s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	16m 10s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	4m 28s		trunk passed
+1 :green_heart:	mvnsite	2m 42s		trunk passed
+1 :green_heart:	javadoc	1m 56s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 44s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 53s		trunk passed
+1 :green_heart:	shadedclient	34m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 34s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	1m 28s		the patch passed
+1 :green_heart:	compile	16m 55s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	16m 55s		the patch passed
+1 :green_heart:	compile	16m 9s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	16m 9s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	4m 29s	/results-checkstyle-root.txt	root: The patch generated 1 new + 116 unchanged - 0 fixed = 117 total (was 116)
+1 :green_heart:	mvnsite	2m 38s		the patch passed
+1 :green_heart:	javadoc	1m 51s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 46s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	4m 15s		the patch passed
+1 :green_heart:	shadedclient	34m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	19m 49s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 55s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	1m 5s		The patch does not generate ASF License warnings.
		242m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6789
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 7a1904c6c487 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / c2168908fd65ec5be399289e425c537f6ba3a63c
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/testReport/
Max. process+thread count	1379 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 1s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 :ok:	mvndep	14m 56s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	32m 9s		trunk passed
+1 :green_heart:	compile	16m 50s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	15m 32s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	4m 17s		trunk passed
+1 :green_heart:	mvnsite	2m 32s		trunk passed
+1 :green_heart:	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 46s		trunk passed
+1 :green_heart:	shadedclient	34m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 32s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	1m 22s		the patch passed
+1 :green_heart:	compile	16m 21s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	16m 21s		the patch passed
+1 :green_heart:	compile	15m 23s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	15m 23s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	4m 14s		the patch passed
+1 :green_heart:	mvnsite	2m 30s		the patch passed
+1 :green_heart:	javadoc	1m 43s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	4m 9s		the patch passed
+1 :green_heart:	shadedclient	33m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	19m 22s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 49s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	1m 0s		The patch does not generate ASF License warnings.
		236m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/17/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6789
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 7722d8fce459 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d55ffebcba7f6c64f1d51220b3e9a789069958cc
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/17/testReport/
Max. process+thread count	1309 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/17/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

@mukund-thakur thanks, altered the tests and tried to clarify the immutability.

One final thought: should we have a default list of enum values to set if, when parsing the config string, the string isn't found?

Currently if nothing is found we return the empty set.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 :ok:	mvndep	14m 31s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	32m 9s		trunk passed
+1 :green_heart:	compile	16m 59s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	15m 48s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	4m 17s		trunk passed
+1 :green_heart:	mvnsite	2m 30s		trunk passed
+1 :green_heart:	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 46s		trunk passed
+1 :green_heart:	shadedclient	33m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 33s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	1m 23s		the patch passed
+1 :green_heart:	compile	16m 19s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	16m 19s		the patch passed
+1 :green_heart:	compile	15m 25s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	15m 25s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	4m 6s		the patch passed
+1 :green_heart:	mvnsite	2m 30s		the patch passed
+1 :green_heart:	javadoc	1m 45s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 34s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	4m 8s		the patch passed
+1 :green_heart:	shadedclient	34m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	19m 19s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 48s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	1m 0s		The patch does not generate ASF License warnings.
		236m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/18/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6789
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 4c294b2bd8b6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / f330597d6801db879011a95793e8555705e6e355
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/18/testReport/
Max. process+thread count	2014 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/18/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[mukund-thakur]

@mukund-thakur thanks, altered the tests and tried to clarify the immutability.

One final thought: should we have a default list of enum values to set if, when parsing the config string, the string isn't found?

Currently if nothing is found we return the empty set.

I am good with an empty set. What is the default value you are thinking? create is already getting added based on the create perf flag explicitly.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 1s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 :ok:	mvndep	14m 16s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	20m 10s		trunk passed
+1 :green_heart:	compile	9m 0s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	8m 5s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	2m 6s		trunk passed
+1 :green_heart:	mvnsite	1m 35s		trunk passed
+1 :green_heart:	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 9s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	2m 27s		trunk passed
+1 :green_heart:	shadedclient	21m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 20s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	0m 51s		the patch passed
+1 :green_heart:	compile	8m 35s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	8m 35s		the patch passed
+1 :green_heart:	compile	8m 7s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	8m 7s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	2m 4s		the patch passed
+1 :green_heart:	mvnsite	1m 36s		the patch passed
+1 :green_heart:	javadoc	1m 12s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 9s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	2m 35s		the patch passed
+1 :green_heart:	shadedclient	20m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	16m 37s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 10s		hadoop-aws in the patch passed.
+1 :green_heart:	asflicense	0m 44s		The patch does not generate ASF License warnings.
		150m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/19/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6789
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 0378336d9b15 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / ce8dac0c544508a002790d4a8c5e09251b6500ff
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/19/testReport/
Max. process+thread count	1282 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/19/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

ok, everything is addressed, going to merge to the branches