HADOOP-19073 WASB: Fix connection leak in FolderRenamePending

[xuzifu666]

Description of PR

Currently connection leak in FolderRenamePending in getting bytes. the pr aim to fix it

How was this patch tested?

For code changes:

• [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[xuzifu666]

@ayushtkn @slfan1989 PTAL

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	42m 6s		trunk passed
+1 :green_heart:	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 31s		trunk passed
+1 :green_heart:	mvnsite	0m 39s		trunk passed
+1 :green_heart:	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 4s		trunk passed
+1 :green_heart:	shadedclient	33m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 28s		the patch passed
+1 :green_heart:	compile	0m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 28s		the patch passed
+1 :green_heart:	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 26s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 19s		the patch passed
+1 :green_heart:	mvnsite	0m 30s		the patch passed
+1 :green_heart:	javadoc	0m 25s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 2s		the patch passed
+1 :green_heart:	shadedclient	32m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	1m 57s		hadoop-azure in the patch passed.
+1 :green_heart:	asflicense	0m 35s		The patch does not generate ASF License warnings.
		123m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6534
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ac3427661d95 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 649c40fe96cc84f17ff0d97949fc2b55b46772e9
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/1/testReport/
Max. process+thread count	700 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	12m 56s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	42m 5s		trunk passed
+1 :green_heart:	compile	0m 36s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	checkstyle	0m 32s		trunk passed
+1 :green_heart:	mvnsite	0m 38s		trunk passed
+1 :green_heart:	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 3s		trunk passed
+1 :green_heart:	shadedclient	31m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	0m 26s		the patch passed
+1 :green_heart:	compile	0m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javac	0m 28s		the patch passed
+1 :green_heart:	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	javac	0m 25s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 18s		the patch passed
+1 :green_heart:	mvnsite	0m 28s		the patch passed
+1 :green_heart:	javadoc	0m 25s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart:	spotbugs	1m 3s		the patch passed
+1 :green_heart:	shadedclient	32m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	2m 15s		hadoop-azure in the patch passed.
+1 :green_heart:	asflicense	0m 35s		The patch does not generate ASF License warnings.
		134m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6534
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5eae444d189b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 649c40fe96cc84f17ff0d97949fc2b55b46772e9
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/2/testReport/
Max. process+thread count	624 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[zhangshuyan0]

Well, this may not belong to the HDFS module.

[slfan1989]

Well, this may not belong to the HDFS module.

@zhangshuyan0 Thank you for your review! but I think this PR needs Steve’s help to review it.

@xuzifu666 Thanks for the contribution! we need to wait for the results of Steve's review.

[xuzifu666]

Well, this may not belong to the HDFS module.

@zhangshuyan0 Thank you for your review! but I think this PR needs Steve’s help to review it.

@xuzifu666 Thanks for the contribution! we need to wait for the results of Steve's review.

OK， @steveloughran could you give the final review? or @slfan1989 could you help to ping the commiters related to the pr to review it？Thanks ~

[steveloughran]

moved to hadoop; azure component.

now, the bad news: as it goes near a cloud store, you have to run the entire hadoop-azure tests for the wasb component, and tell us which region you tested against.

https://hadoop.apache.org/docs/stable/hadoop-azure/testing_azure.html

sorry, but yetus doesn't have any credentials. It doesn't take long and is designed to clean up afterwards

[xuzifu666]

moved to hadoop; azure component.

now, the bad news: as it goes near a cloud store, you have to run the entire hadoop-azure tests for the wasb component, and tell us which region you tested against.

https://hadoop.apache.org/docs/stable/hadoop-azure/testing_azure.html

sorry, but yetus doesn't have any credentials. It doesn't take long and is designed to clean up afterwards

OK，anything else me to do for the pull request？@steveloughran

[steveloughran]

@xuzifu666 yes.

1. Which azure region did you run all the wasb tests against
2. and what were your maven command line arguments used?
3. did any tests fail?

[xuzifu666]

@xuzifu666 yes.

1. Which azure region did you run all the wasb tests against
2. and what were your maven command line arguments used?
3. did any tests fail?

@steveloughran Hi，seems network connection available，test is hard for me to execute，but the pr may be a obvious connection leak，could you give a help to confirm it？Thank any way~

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 :green_heart:	dupname	0m 00s		No case conflicting files found.
+0 :ok:	spotbugs	0m 00s		spotbugs executables are not available.
+0 :ok:	codespell	0m 01s		codespell was not available.
+0 :ok:	detsecrets	0m 01s		detect-secrets was not available.
+1 :green_heart:	@author	0m 00s		The patch does not contain any @author tags.
-1 :x:	test4tests	0m 00s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	86m 45s		trunk passed
+1 :green_heart:	compile	4m 35s		trunk passed
+1 :green_heart:	checkstyle	4m 20s		trunk passed
+1 :green_heart:	mvnsite	4m 45s		trunk passed
+1 :green_heart:	javadoc	4m 26s		trunk passed
+1 :green_heart:	shadedclient	140m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	2m 33s		the patch passed
+1 :green_heart:	compile	2m 06s		the patch passed
+1 :green_heart:	javac	2m 06s		the patch passed
+1 :green_heart:	blanks	0m 01s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	1m 56s		the patch passed
+1 :green_heart:	mvnsite	2m 13s		the patch passed
+1 :green_heart:	javadoc	2m 01s		the patch passed
+1 :green_heart:	shadedclient	150m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	asflicense	5m 41s		The patch does not generate ASF License warnings.
		399m 51s

Subsystem	Report/Notes
GITHUB PR	https://github.com/apache/hadoop/pull/6534
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	MINGW64_NT-10.0-17763 2ea128c33854 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	trunk / 649c40fe96cc84f17ff0d97949fc2b55b46772e9
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6534/1/testReport/
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6534/1/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[xuzifu666]

@xuzifu666 yes.

1. Which azure region did you run all the wasb tests against
2. and what were your maven command line arguments used?
3. did any tests fail?

@steveloughran Hi，seems network connection available，test is hard for me to execute，but the pr may be a obvious connection leak，could you give a help to confirm it？Thank any way~ cc@steveloughran

[xuzifu666]

@steveloughran could you give the final review or close the pr? Thanks

[steveloughran]

let me try and test this myself.

[steveloughran]

(which I can't do today as both my hadoop source trees are testing my "support parallel tests against the same s3 bucket" right now...

[xuzifu666]

let me try and test this myself.

OK，Thanks，currently it can be work now？ @steveloughran

[steveloughran]

I've been away for a week and am catching up, so not tested yet, sorry...

[steveloughran]

@xuzifu666 before I merge, what name do you want to be credited with in the patch...your github account doesn't have one.

[steveloughran]

thanks, merged to trunk and updated the JIRA.

If you do a cherrypick of this commit and submit as a PR against branch-3.4 I'll merge it there too, once yetus is happy