HADOOP-16649. hadoop_add_param function : change regexp test by iterative equality test

[Taccart]

The bug HADOOP-16649 is due to a regexp test : hadoop-azure is detected as already present when hadoop-azure-datalake is present. This PR contains an update for hadoop_add_param function to address the problem, and the corresponding bats test.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	29m 8s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	45m 24s		trunk passed
+1 :green_heart:	mvnsite	1m 32s		trunk passed
+1 :green_heart:	shadedclient	15m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 56s		the patch passed
+1 :green_heart:	mvnsite	1m 13s		the patch passed
-1 :x:	shellcheck	0m 5s	/diff-patch-shellcheck.txt	The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20)
+1 :green_heart:	shelldocs	0m 17s		The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
+1 :green_heart:	whitespace	0m 0s		The patch has no whitespace issues.
-1 :x:	shadedclient	3m 24s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	3m 24s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
+1 :green_heart:	asflicense	0m 30s		The patch does not generate ASF License warnings.
		102m 50s

Reason	Tests
Failed TAP tests	hadoop_finalize_hadoop_heap.bats.tap

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/2385
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs
uname	Linux 61c5a95c2099 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 4c651103f2a
TAP logs	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/artifact/out/patch-hadoop-common-project_hadoop-common.tap
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	30m 51s		trunk passed
+1 :green_heart:	mvnsite	1m 25s		trunk passed
+1 :green_heart:	shadedclient	15m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 55s		the patch passed
+1 :green_heart:	mvnsite	1m 11s		the patch passed
-1 :x:	shellcheck	0m 4s	/diff-patch-shellcheck.txt	The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20)
+1 :green_heart:	shelldocs	0m 17s		The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
+1 :green_heart:	whitespace	0m 0s		The patch has no whitespace issues.
+1 :green_heart:	shadedclient	14m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	3m 34s		hadoop-common in the patch passed.
+1 :green_heart:	asflicense	0m 38s		The patch does not generate ASF License warnings.
		71m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/2385
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs
uname	Linux 188c2fbdc0c5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / a2e7a073f03
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/2/testReport/
Max. process+thread count	449 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/2/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 28s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	30m 29s		trunk passed
+1 :green_heart:	mvnsite	1m 25s		trunk passed
+1 :green_heart:	shadedclient	14m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 54s		the patch passed
+1 :green_heart:	mvnsite	1m 11s		the patch passed
-1 :x:	shellcheck	0m 4s	/diff-patch-shellcheck.txt	The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20)
+1 :green_heart:	shelldocs	0m 16s		The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
+1 :green_heart:	whitespace	0m 0s		The patch has no whitespace issues.
+1 :green_heart:	shadedclient	14m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	3m 33s		hadoop-common in the patch passed.
+1 :green_heart:	asflicense	0m 37s		The patch does not generate ASF License warnings.
		70m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/3/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/2385
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs
uname	Linux 2b316f3fc4c0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / a2e7a073f03
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/3/testReport/
Max. process+thread count	444 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/3/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	29m 27s		trunk passed
+1 :green_heart:	mvnsite	1m 32s		trunk passed
+1 :green_heart:	shadedclient	16m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 5s		the patch passed
+1 :green_heart:	mvnsite	1m 19s		the patch passed
-1 :x:	shellcheck	0m 6s	/diff-patch-shellcheck.txt	The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20)
+1 :green_heart:	shelldocs	0m 15s		The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
+1 :green_heart:	whitespace	0m 0s		The patch has no whitespace issues.
+1 :green_heart:	shadedclient	15m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	3m 15s		hadoop-common in the patch passed.
+1 :green_heart:	asflicense	0m 35s		The patch does not generate ASF License warnings.
		71m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/4/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/2385
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs
uname	Linux b7d5a4f95d92 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 88a9f42f320
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/4/testReport/
Max. process+thread count	439 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/4/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	29m 22s		trunk passed
+1 :green_heart:	mvnsite	1m 24s		trunk passed
+1 :green_heart:	shadedclient	15m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 56s		the patch passed
+1 :green_heart:	mvnsite	1m 10s		the patch passed
+1 :green_heart:	shellcheck	0m 5s		There were no new shellcheck issues.
+1 :green_heart:	shelldocs	0m 17s		The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
+1 :green_heart:	whitespace	0m 0s		The patch has no whitespace issues.
+1 :green_heart:	shadedclient	14m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	3m 32s		hadoop-common in the patch passed.
+1 :green_heart:	asflicense	0m 36s		The patch does not generate ASF License warnings.
		70m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/5/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/2385
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs
uname	Linux e8f88d772cb1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 88a9f42f320
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/5/testReport/
Max. process+thread count	424 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/5/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[aw-was-here]

Just to save everyone a lot of time and suffering:

This approach will break a lot of things in very unexpected ways (doing a search on everywhere hadoop_add_params is called should make this clear). hadoop_add_param was specifically built for partial matches because the HADOOP_OPTS command line can't really do exact matches and this was a quick way to prevent duplicate options. The unit test failure in hadoop_finalize_hadoop_heap was intended to provide a hint that "yar they be dragons here." I should have written better tests, but given it took like 2 years just to get most of this code in over the total @#$@#$ that was in hadoop 2.x ...

When I wrote the code originally, we didn't have a need for exact matches anywhere (HADOOP_OPTIONAL_TOOLS wasn't written yet). It was written and committed to 3.x. Then the HADOOP_OPTIONAL_TOOLS code was written but that would be the only place where an exact match would be useful and we didn't have any sooo... I just re-used hadoop_add_param with the (clearly faulty) assumption that people would test their code on Hadoop 3.x. But since the azure team didn't bother to test with hadoop 3.x until it was too late... At this point, I was getting tired of the Hadoop politics and bailed, leaving this furball hanging around.

Anyway, the real fix for this is to convert HADOOP_OPTIONAL_TOOLS to an array and then do an exact match, looping over the array. I think there is code to do that now. Might need some new helper code to do comma-delimited -> array but that shouldn't be hard.

[ThAccart]

Hi thanks for your comment.

The new code keeps the privious pattern matching test (sorry for the misleading title) : it will accept a new param only if it is not found using the previous pattern testing and not found using an equality testing (to address the hadoop-azure hadoop-azure-datalake)

I also added a specific test for the pattern matching :

@test "hadoop_add_param (HADOOP-16649 c )" { hadoop_add_param testvar Xmx -Xmx2048 hadoop_add_param testvar Xmx -Xmx128 hadoop_add_param testvar Xms -Xms32 echo ">${testvar}<" [ "${testvar}" = "-Xmx2048 -Xms32" ] }

[ThAccart]

Sad to see no evolution of this PR since 1 month.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	shelldocs	0m 0s		Shelldocs was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	41m 8s		trunk passed
+1 :green_heart:	mvnsite	1m 38s		trunk passed
+1 :green_heart:	shadedclient	22m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 4s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	mvnsite	1m 24s		the patch passed
+1 :green_heart:	shellcheck	0m 4s		No new issues.
+1 :green_heart:	shadedclient	22m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	1m 50s		hadoop-common in the patch passed.
+1 :green_heart:	asflicense	0m 44s		The patch does not generate ASF License warnings.
		95m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/2385
Optional Tests	dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs
uname	Linux fe58257e8835 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / e595876ebae392e28357903cbe53af666b99d904
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/testReport/
Max. process+thread count	534 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2385/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.