HDFS-17524. OIV: add Transformed processor which reconstructs an fsimage from another fsimage file

[XbaoWu]

Description of PR

Hadoop 3.2 introduced optimization features for HDFS StringTable (https://github.com/apache/hadoop/commit/b60ca37914b22550e3630fa02742d40697decb31), It resulted in lower versions of Hadoop upgraded to 3.2 and later versions not supporting downgrade operations. Therefore, we have added Transformed Processor to support downgrading. In the upgrade with large version span, it avoids the low version can not read the fs-image generated by the 3.2 and later version, which support for the 3.2 and later version to be downgraded to the low version.

For more information about this PR, please refer to the following issue： HDFS-17524 OIV: add Transformed processor which reconstructs an fsimage from another fsimage file.

How was this patch tested?

add TestOfflineImageViewer.testTransformedWriter() add TestOfflineImageViewer.testTransformedWriterWrongLayoutVersion() add TestOfflineImageViewer.testTransformedWriterReasonableLayoutVersion()

For code changes:

• [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 :green_heart:	dupname	0m 01s		No case conflicting files found.
+0 :ok:	spotbugs	0m 01s		spotbugs executables are not available.
+0 :ok:	codespell	0m 01s		codespell was not available.
+0 :ok:	detsecrets	0m 01s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 01s		markdownlint was not available.
+1 :green_heart:	@author	0m 00s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 00s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	91m 53s		trunk passed
+1 :green_heart:	compile	6m 45s		trunk passed
+1 :green_heart:	checkstyle	5m 11s		trunk passed
+1 :green_heart:	mvnsite	7m 07s		trunk passed
+1 :green_heart:	javadoc	6m 13s		trunk passed
+1 :green_heart:	shadedclient	155m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	4m 49s		the patch passed
+1 :green_heart:	compile	3m 44s		the patch passed
+1 :green_heart:	javac	3m 44s		the patch passed
-1 :x:	blanks	0m 01s	/blanks-eol.txt	The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 :green_heart:	checkstyle	2m 28s		the patch passed
+1 :green_heart:	mvnsite	4m 21s		the patch passed
+1 :green_heart:	javadoc	3m 52s		the patch passed
+1 :green_heart:	shadedclient	163m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	asflicense	5m 43s		The patch does not generate ASF License warnings.
		439m 29s

Subsystem	Report/Notes
GITHUB PR	https://github.com/apache/hadoop/pull/6820
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	MINGW64_NT-10.0-17763 cbd8002785da 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	trunk / 6fe2b6fe1df0eb5c1c3958dece6a370b698e2721
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6820/2/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6820/2/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
-1 :x:	mvninstall	1m 59s	/branch-mvninstall-root.txt	root in trunk failed.
-1 :x:	compile	0m 21s	/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	compile	0m 20s	/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
-0 :warning:	checkstyle	0m 19s	/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	The patch fails to run checkstyle in hadoop-hdfs
-1 :x:	mvnsite	0m 22s	/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in trunk failed.
-1 :x:	javadoc	0m 21s	/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	javadoc	0m 22s	/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
-1 :x:	spotbugs	0m 22s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in trunk failed.
+1 :green_heart:	shadedclient	2m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 :x:	mvninstall	0m 22s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 :x:	compile	0m 22s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	javac	0m 22s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	compile	0m 23s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
-1 :x:	javac	0m 23s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	2m 21s	/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	The patch fails to run checkstyle in hadoop-hdfs
-1 :x:	mvnsite	0m 42s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 :x:	javadoc	0m 22s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
-1 :x:	javadoc	0m 39s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt	hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.
-1 :x:	spotbugs	0m 22s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 :x:	shadedclient	9m 4s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	0m 17s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+0 :ok:	asflicense	0m 22s		ASF License check generated no output?
		20m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6820/4/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6820
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 1c835e6a3fd6 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0ab78d70a4518262f7d5fe759b9e1575bc087e33
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6820/4/testReport/
Max. process+thread count	56 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6820/4/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 :green_heart:	dupname	0m 02s		No case conflicting files found.
+0 :ok:	spotbugs	0m 01s		spotbugs executables are not available.
+0 :ok:	codespell	0m 01s		codespell was not available.
+0 :ok:	detsecrets	0m 01s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 01s		markdownlint was not available.
+1 :green_heart:	@author	0m 01s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 00s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	120m 10s		trunk passed
+1 :green_heart:	compile	8m 13s		trunk passed
+1 :green_heart:	checkstyle	6m 39s		trunk passed
+1 :green_heart:	mvnsite	8m 43s		trunk passed
+1 :green_heart:	javadoc	8m 59s		trunk passed
+1 :green_heart:	shadedclient	187m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	6m 44s		the patch passed
+1 :green_heart:	compile	5m 09s		the patch passed
+1 :green_heart:	javac	5m 09s		the patch passed
+1 :green_heart:	blanks	0m 01s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	3m 20s		the patch passed
+1 :green_heart:	mvnsite	5m 47s		the patch passed
+1 :green_heart:	javadoc	5m 02s		the patch passed
+1 :green_heart:	shadedclient	201m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	asflicense	8m 11s		The patch does not generate ASF License warnings.
		548m 31s

Subsystem	Report/Notes
GITHUB PR	https://github.com/apache/hadoop/pull/6820
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	MINGW64_NT-10.0-17763 45ce868edf92 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	trunk / 0ab78d70a4518262f7d5fe759b9e1575bc087e33
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6820/4/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6820/4/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[github-actions[bot]]

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.