HADOOP-18013. ABFS: add cloud trash policy with per-schema policy selection

[steveloughran]

New trash policies, and a schema specific trash policy set by

fs.SCHEMA.trash.policy.

This lets clusters declare different policies for different stores in the same cluster.

• CloudTrashPolicy: for abfs with rename failure resilience and auto cleanup of old checkpoints.
• DeleteFilesTrashPolicy: for versioned s3 buckets; delete the files.

How was this patch tested?

what do you mean, tested?

For code changes:

• [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[steveloughran]

Still a WiP but should interest people working with abfs/gcs and to a lesser degree s3a

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 1s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
-1 :x:	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 :ok:	mvndep	15m 15s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	29m 18s		trunk passed
+1 :green_heart:	compile	25m 25s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	compile	21m 37s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	checkstyle	4m 14s		trunk passed
+1 :green_heart:	mvnsite	3m 17s		trunk passed
+1 :green_heart:	javadoc	2m 44s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart:	javadoc	2m 6s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart:	spotbugs	4m 55s		trunk passed
+1 :green_heart:	shadedclient	22m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 31s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	1m 44s		the patch passed
+1 :green_heart:	compile	24m 9s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
-1 :x:	javac	24m 9s	/results-compile-javac-root-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt	root-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 5 new + 2850 unchanged - 4 fixed = 2855 total (was 2854)
+1 :green_heart:	compile	21m 23s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 :x:	javac	21m 23s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt	root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 4 new + 2649 unchanged - 3 fixed = 2653 total (was 2652)
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	4m 6s	/results-checkstyle-root.txt	root: The patch generated 5 new + 73 unchanged - 4 fixed = 78 total (was 77)
+1 :green_heart:	mvnsite	3m 8s		the patch passed
-1 :x:	javadoc	1m 23s	/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt	hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 :green_heart:	javadoc	2m 8s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 :x:	spotbugs	3m 3s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 :green_heart:	shadedclient	22m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	19m 12s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 34s		hadoop-azure in the patch passed.
+1 :green_heart:	asflicense	1m 24s		The patch does not generate ASF License warnings.
		246m 20s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Dead store to dir in org.apache.hadoop.fs.TrashPolicyDefault.deleteCheckpoint(Path) At TrashPolicyDefault.java:org.apache.hadoop.fs.TrashPolicyDefault.deleteCheckpoint(Path) At TrashPolicyDefault.java:[line 412]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4729/3/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/4729
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0cebf7eb06c2 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 5a16dea486e9b498777b6330b075d18063d582d3
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4729/3/testReport/
Max. process+thread count	2928 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4729/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.