hbase-operator-tools icon indicating copy to clipboard operation
hbase-operator-tools copied to clipboard
verified publisher icon apache

HBASE-25161 [hbase-meta-repair] Add tool for repairing metadata.

Open VictorZeng opened this issue 5 years ago • 18 comments

fix HBASE-25161

VictorZeng avatar Oct 07 '20 09:10 VictorZeng

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 15s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 45s master passed
+1 :green_heart: compile 0m 9s master passed
+1 :green_heart: checkstyle 0m 6s master passed
+1 :green_heart: javadoc 0m 7s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 7s Maven dependency ordering for patch
+1 :green_heart: mvninstall 0m 24s the patch passed
+1 :green_heart: compile 0m 17s the patch passed
+1 :green_heart: javac 0m 17s the patch passed
-1 :x: checkstyle 0m 5s hbase-meta-repair: The patch generated 193 new + 0 unchanged - 0 fixed = 193 total (was 0)
-1 :x: checkstyle 0m 6s root: The patch generated 193 new + 0 unchanged - 0 fixed = 193 total (was 0)
-1 :x: whitespace 0m 0s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 :green_heart: xml 0m 1s The patch has no ill-formed XML file.
+1 :green_heart: javadoc 0m 10s the patch passed
_ Other Tests _
-1 :x: unit 10m 55s hbase-meta-repair in the patch failed.
-1 :x: unit 18m 30s root in the patch failed.
-1 :x: asflicense 0m 9s The patch generated 1 ASF License warnings.
33m 19s
Reason Tests
Failed junit tests hbase.repair.TestMetaRepair
hbase.TestRegionsMerger
hbase.repair.TestMetaRepair
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 2926be98c042 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 5cdc0e2
Default Java Oracle Corporation-1.8.0_265-b01
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/diff-checkstyle-hbase-meta-repair.txt
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/diff-checkstyle-root.txt
whitespace https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/whitespace-eol.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-unit-hbase-meta-repair.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/testReport/
asflicense https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-asflicense-problems.txt
Max. process+thread count 941 (vs. ulimit of 1000)
modules C: hbase-meta-repair . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 07 '20 09:10 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 41s master passed
+1 :green_heart: compile 0m 9s master passed
+1 :green_heart: checkstyle 0m 6s master passed
+1 :green_heart: javadoc 0m 8s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 7s Maven dependency ordering for patch
+1 :green_heart: mvninstall 0m 25s the patch passed
+1 :green_heart: compile 0m 17s the patch passed
+1 :green_heart: javac 0m 17s the patch passed
-1 :x: checkstyle 0m 4s hbase-meta-repair: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
-1 :x: checkstyle 0m 4s root: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 1s The patch has no ill-formed XML file.
+1 :green_heart: javadoc 0m 10s the patch passed
_ Other Tests _
-1 :x: unit 0m 31s hbase-meta-repair in the patch failed.
-1 :x: unit 6m 44s root in the patch failed.
+1 :green_heart: asflicense 0m 9s The patch does not generate ASF License warnings.
10m 21s
Reason Tests
Failed junit tests hbase.repair.TestMetaRepair
hbase.TestRegionsMerger
hbase.repair.TestMetaRepair
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 02832fc2b867 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 5cdc0e2
Default Java Oracle Corporation-1.8.0_265-b01
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/artifact/yetus-precommit-check/output/diff-checkstyle-hbase-meta-repair.txt
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/artifact/yetus-precommit-check/output/diff-checkstyle-root.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/artifact/yetus-precommit-check/output/patch-unit-hbase-meta-repair.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/artifact/yetus-precommit-check/output/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/testReport/
Max. process+thread count 939 (vs. ulimit of 1000)
modules C: hbase-meta-repair . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/2/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 07 '20 10:10 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 48s master passed
+1 :green_heart: compile 0m 10s master passed
+1 :green_heart: checkstyle 0m 9s master passed
+1 :green_heart: javadoc 0m 7s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 7s Maven dependency ordering for patch
+1 :green_heart: mvninstall 0m 42s the patch passed
+1 :green_heart: compile 0m 17s the patch passed
+1 :green_heart: javac 0m 17s the patch passed
-1 :x: checkstyle 0m 3s hbase-meta-repair: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
-1 :x: checkstyle 0m 4s root: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 2s The patch has no ill-formed XML file.
+1 :green_heart: javadoc 0m 9s the patch passed
_ Other Tests _
-1 :x: unit 0m 31s hbase-meta-repair in the patch failed.
-1 :x: unit 26m 49s root in the patch failed.
+1 :green_heart: asflicense 0m 10s The patch does not generate ASF License warnings.
31m 0s
Reason Tests
Failed junit tests hbase.repair.TestMetaRepair
hbase.repair.TestMetaRepair
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 90555a1d353a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 5cdc0e2
Default Java Oracle Corporation-1.8.0_265-b01
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/artifact/yetus-precommit-check/output/diff-checkstyle-hbase-meta-repair.txt
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/artifact/yetus-precommit-check/output/diff-checkstyle-root.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/artifact/yetus-precommit-check/output/patch-unit-hbase-meta-repair.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/artifact/yetus-precommit-check/output/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/testReport/
Max. process+thread count 935 (vs. ulimit of 1000)
modules C: hbase-meta-repair . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/3/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 07 '20 11:10 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 42s master passed
+1 :green_heart: compile 0m 9s master passed
+1 :green_heart: checkstyle 0m 6s master passed
+1 :green_heart: javadoc 0m 7s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 8s Maven dependency ordering for patch
+1 :green_heart: mvninstall 0m 25s the patch passed
+1 :green_heart: compile 0m 17s the patch passed
+1 :green_heart: javac 0m 17s the patch passed
-1 :x: checkstyle 0m 3s hbase-meta-repair: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
-1 :x: checkstyle 0m 5s root: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 2s The patch has no ill-formed XML file.
+1 :green_heart: javadoc 0m 10s the patch passed
_ Other Tests _
-1 :x: unit 3m 19s hbase-meta-repair in the patch passed.
-1 :x: unit 17m 40s root in the patch failed.
+1 :green_heart: asflicense 0m 9s The patch does not generate ASF License warnings.
24m 6s
Reason Tests
Failed junit tests hbase.repair.TestMetaRepair
hbase.TestRegionsMerger
hbase.repair.TestMetaRepair
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux a0c3cca40594 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 5cdc0e2
Default Java Oracle Corporation-1.8.0_265-b01
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/artifact/yetus-precommit-check/output/diff-checkstyle-hbase-meta-repair.txt
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/artifact/yetus-precommit-check/output/diff-checkstyle-root.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/artifact/yetus-precommit-check/output/patch-unit-hbase-meta-repair.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/artifact/yetus-precommit-check/output/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/testReport/
Max. process+thread count 942 (vs. ulimit of 1000)
modules C: hbase-meta-repair . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/4/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 07 '20 11:10 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 44s master passed
+1 :green_heart: compile 0m 10s master passed
+1 :green_heart: checkstyle 0m 6s master passed
+1 :green_heart: javadoc 0m 7s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 8s Maven dependency ordering for patch
+1 :green_heart: mvninstall 0m 25s the patch passed
+1 :green_heart: compile 0m 17s the patch passed
+1 :green_heart: javac 0m 17s the patch passed
+1 :green_heart: checkstyle 0m 8s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 2s The patch has no ill-formed XML file.
+1 :green_heart: javadoc 0m 10s the patch passed
_ Other Tests _
+1 :green_heart: unit 0m 46s hbase-meta-repair in the patch passed.
-1 :x: unit 8m 19s root in the patch failed.
+1 :green_heart: asflicense 0m 9s The patch does not generate ASF License warnings.
12m 16s
Reason Tests
Failed junit tests hbase.TestRegionsMerger
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/5/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 2fdc34118b31 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 5cdc0e2
Default Java Oracle Corporation-1.8.0_265-b01
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/5/artifact/yetus-precommit-check/output/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/5/testReport/
Max. process+thread count 942 (vs. ulimit of 1000)
modules C: hbase-meta-repair . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/5/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 07 '20 12:10 Apache-HBase

This testcase for RegionsMerger run failed:

[INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hbase.TestRegionsMerger [ERROR] Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 102.515 s <<< FAILURE! - in org.apache.hbase.TestRegionsMerger [ERROR] testMergeRegionsInvalidParams(org.apache.hbase.TestRegionsMerger) Time elapsed: 0.422 s <<< ERROR! java.io.IOException: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread ......

VictorZeng avatar Oct 07 '20 12:10 VictorZeng

@wchevreuil There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions. If users use Hbase 2.1.0 version, you can use this tool to repair metadata.

VictorZeng avatar Oct 07 '20 12:10 VictorZeng

Have you tried HBCK2 addFsRegionsMissingInMeta and extraRegionsInMeta functions? I believe those do the same as what you are proposing here, in other words, use hdfs info as the source of truth to rebuild meta.

This is a fallback tool to reconstruct the meta region when users are deploying a 2.x version without HBCK2 support. I think it is useful for our users?

Apache9 avatar Oct 07 '20 12:10 Apache9

There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions. If users use Hbase 2.1.0 version, you can use this tool to repair metadata.

Have you tried use these functions and got some error? If so, we should fix that, as neither addFsRegionsMissingInMeta nor extraRegionsInMeta rely on HbckService, they perform fixes on their own, thus, there should be no hbase version validation when using these two functions, and they should be compatible with the above versions.

wchevreuil avatar Oct 08 '20 09:10 wchevreuil

There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions. If users use Hbase 2.1.0 version, you can use this tool to repair metadata.

Have you tried use these functions and got some error? If so, we should fix that, as neither addFsRegionsMissingInMeta nor extraRegionsInMeta rely on HbckService, they perform fixes on their own, thus, there should be no hbase version validation when using these two functions, and they should be compatible with the above versions.

Hi wchevreuil, although both addFsRegionsMissingInMeta and extraRegionsInMeta do not rely on HbckService, but assigns command rely on HbckService... So I think this tool is very useful.

VictorZeng avatar Oct 08 '20 10:10 VictorZeng

There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions. If users use Hbase 2.1.0 version, you can use this tool to repair metadata.

Have you tried use these functions and got some error? If so, we should fix that, as neither addFsRegionsMissingInMeta nor extraRegionsInMeta rely on HbckService, they perform fixes on their own, thus, there should be no hbase version validation when using these two functions, and they should be compatible with the above versions.

Hi wchevreuil, although both addFsRegionsMissingInMeta and extraRegionsInMeta do not rely on HbckService, but assigns command rely on HbckService... So I think this tool is very useful.

Hi wchevreuil, although both addFsRegionsMissingInMeta and extraRegionsInMeta do not rely on HbckService, but assigns command rely on HbckService... So I think this tool is very useful.

Then the only extra bit is a bulk assignment? Could it reuse addFsRegionsMissingInMeta and extraRegionsInMeta, instead of providing extra lines of code that do same as the already existing methods? I do agree the command proposed here is very convenient, but we should re-use already existing code as much as possible.

wchevreuil avatar Oct 08 '20 15:10 wchevreuil

There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions. If users use Hbase 2.1.0 version, you can use this tool to repair metadata.

Have you tried use these functions and got some error? If so, we should fix that, as neither addFsRegionsMissingInMeta nor extraRegionsInMeta rely on HbckService, they perform fixes on their own, thus, there should be no hbase version validation when using these two functions, and they should be compatible with the above versions.

Hi wchevreuil, although both addFsRegionsMissingInMeta and extraRegionsInMeta do not rely on HbckService, but assigns command rely on HbckService... So I think this tool is very useful.

Hi wchevreuil, although both addFsRegionsMissingInMeta and extraRegionsInMeta do not rely on HbckService, but assigns command rely on HbckService... So I think this tool is very useful.

Then the only extra bit is a bulk assignment? Could it reuse addFsRegionsMissingInMeta and extraRegionsInMeta, instead of providing extra lines of code that do same as the already existing methods? I do agree the command proposed here is very convenient, but we should re-use already existing code as much as possible.

This tool function is more similar to the hbck1 fixMeta command, it can repair the metadata when users are deploying a 2.x version without HBCK2 support more quickly and conveniently. So it's necessary to provide extra codes. I think it's better to be independent.

VictorZeng avatar Oct 09 '20 02:10 VictorZeng

it can repair the metadata when users are deploying a 2.x version without HBCK2 support more quickly and conveniently.

I agree with that, and that we can do it without the server side support for hbck2

So it's necessary to provide extra codes. I think it's better to be independent.

We already have functions/methods on hbck2 that perform fixes without relying on server side support for hbck2, so I see no reason to add an entirely extra module whose scope could/should be contained within hbck2. And for this specific solution, I already mentioned some methods that could be reused by some sort of coordination logic that could give a less interactive experience, without the need to duplicate code to do essentially the same thing.

wchevreuil avatar Oct 19 '20 10:10 wchevreuil

@VictorZeng still working on this?

busbey avatar Jan 15 '21 18:01 busbey

@busbey What do you think of megre the Meta Repair Tool into hbase-tools module ?

VictorZeng avatar Feb 07 '21 02:02 VictorZeng

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 9s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 46s master passed
+1 :green_heart: compile 0m 10s master passed
+1 :green_heart: checkstyle 0m 7s master passed
+1 :green_heart: javadoc 0m 8s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 8s Maven dependency ordering for patch
+1 :green_heart: mvninstall 0m 26s the patch passed
+1 :green_heart: compile 0m 19s the patch passed
+1 :green_heart: javac 0m 19s the patch passed
+1 :green_heart: checkstyle 0m 8s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 2s The patch has no ill-formed XML file.
+1 :green_heart: javadoc 0m 11s the patch passed
_ Other Tests _
+1 :green_heart: unit 0m 48s hbase-meta-repair in the patch passed.
+1 :green_heart: unit 6m 14s root in the patch passed.
+1 :green_heart: asflicense 0m 9s The patch does not generate ASF License warnings.
10m 55s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/6/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 8e13243b309e 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 2958eeb
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/6/testReport/
Max. process+thread count 1285 (vs. ulimit of 5000)
modules C: hbase-meta-repair . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/6/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 07 '21 02:02 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 57s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 53s master passed
+1 :green_heart: compile 0m 10s master passed
+1 :green_heart: checkstyle 0m 7s master passed
+1 :green_heart: javadoc 0m 7s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 2s Maven dependency ordering for patch
-1 :x: mvninstall 0m 3s root in the patch failed.
-1 :x: mvninstall 0m 4s hbase-meta-repair in the patch failed.
-1 :x: compile 0m 3s root in the patch failed.
-1 :x: compile 0m 3s hbase-meta-repair in the patch failed.
-1 :x: javac 0m 3s root in the patch failed.
-1 :x: javac 0m 3s hbase-meta-repair in the patch failed.
-1 :x: checkstyle 0m 1s The patch fails to run checkstyle in root
-1 :x: checkstyle 0m 1s The patch fails to run checkstyle in hbase-meta-repair
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 2s The patch has no ill-formed XML file.
-1 :x: javadoc 0m 3s root in the patch failed.
-1 :x: javadoc 0m 3s hbase-meta-repair in the patch failed.
_ Other Tests _
-1 :x: unit 0m 4s root in the patch failed.
-1 :x: unit 0m 3s hbase-meta-repair in the patch failed.
+0 :ok: asflicense 0m 6s ASF License check generated no output?
3m 3s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 46b49e44696a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool maven
git revision master / 8d667b1
Default Java Oracle Corporation-1.8.0_282-b08
mvninstall https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-mvninstall-root.txt
mvninstall https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-mvninstall-hbase-meta-repair.txt
compile https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-root.txt
compile https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-hbase-meta-repair.txt
javac https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-root.txt
javac https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-hbase-meta-repair.txt
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/buildtool-patch-checkstyle-root.txt
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/buildtool-patch-checkstyle-hbase-meta-repair.txt
javadoc https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-javadoc-root.txt
javadoc https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-javadoc-hbase-meta-repair.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-unit-root.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-unit-hbase-meta-repair.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/testReport/
Max. process+thread count 38 (vs. ulimit of 5000)
modules C: . hbase-meta-repair U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 15 '21 02:02 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: markdownlint 0m 0s markdownlint was not available.
+0 :ok: spotbugs 0m 0s spotbugs executables are not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 0m 54s master passed
+1 :green_heart: compile 0m 11s master passed
+1 :green_heart: checkstyle 0m 7s master passed
+1 :green_heart: javadoc 0m 7s master passed
_ Patch Compile Tests _
+0 :ok: mvndep 0m 3s Maven dependency ordering for patch
-1 :x: mvninstall 0m 3s root in the patch failed.
-1 :x: mvninstall 0m 3s hbase-meta-repair in the patch failed.
-1 :x: compile 0m 4s root in the patch failed.
-1 :x: compile 0m 3s hbase-meta-repair in the patch failed.
-1 :x: javac 0m 4s root in the patch failed.
-1 :x: javac 0m 3s hbase-meta-repair in the patch failed.
-1 :x: checkstyle 0m 1s The patch fails to run checkstyle in root
-1 :x: checkstyle 0m 1s The patch fails to run checkstyle in hbase-meta-repair
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: xml 0m 2s The patch has no ill-formed XML file.
-1 :x: javadoc 0m 3s root in the patch failed.
-1 :x: javadoc 0m 3s hbase-meta-repair in the patch failed.
_ Other Tests _
-1 :x: unit 0m 4s root in the patch failed.
-1 :x: unit 0m 3s hbase-meta-repair in the patch failed.
+0 :ok: asflicense 0m 6s ASF License check generated no output?
3m 0s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase-operator-tools/pull/77
Optional Tests dupname asflicense markdownlint javac javadoc unit xml compile spotbugs findbugs checkstyle
uname Linux 6cf55e0ac371 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool maven
git revision master / b7d9fc8
Default Java Oracle Corporation-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-mvninstall-root.txt
mvninstall https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-mvninstall-hbase-meta-repair.txt
compile https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-root.txt
compile https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-hbase-meta-repair.txt
javac https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-compile-hbase-meta-repair.txt
checkstyle https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/buildtool-patch-checkstyle-root.txt
checkstyle https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/buildtool-patch-checkstyle-hbase-meta-repair.txt
javadoc https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-javadoc-root.txt
javadoc https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-javadoc-hbase-meta-repair.txt
unit https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-unit-root.txt
unit https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/artifact/yetus-precommit-check/output/patch-unit-hbase-meta-repair.txt
Test Results https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/testReport/
Max. process+thread count 38 (vs. ulimit of 5000)
modules C: . hbase-meta-repair U: .
Console output https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-77/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Feb 07 '22 04:02 Apache-HBase