apache
HBASE-28919 Soft drop for destructive table actions
Although HFiles are copied to the archive in a destructive schema change, recovery scenarios are not automatic and involve some operator labor to reconstruct the table and re-import the archived data. We can easily prevent the deletion of the HFiles of a deleted table or column family by taking a snapshot of the table immediately prior to any destructive actions. We also set a TTL on the snapshot so housekeeping of unwanted HFiles remains no touch. Because we take a table snapshot all table structure and metadata is also captured and saved so fast recovery is possible, as either a restore from snapshot, or a clone from snapshot to a new table.
Existing site configuration property prerequisites:
hbase.snapshot.enabled = true( default istrue)
New site configuration properties:
hbase.snapshot.before.delete.enabled = true( default isfalse)hbase.snapshot.before.delete.ttl = <integer>, in seconds ( default86400(one day) )
This is a version of an internal change applied to a fork of branch-2.5 that has gone through some validation, ported to the open source master branch. The only significant difference is in master we have a new TruncateRegionProcedure which should also be covered by this feature, and so this required extending the approach to include this procedure and its inheritance hierarchy also.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 51s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 2m 58s | master passed | |
| +1 :green_heart: | compile | 4m 6s | master passed | |
| +1 :green_heart: | checkstyle | 0m 58s | master passed | |
| -1 :x: | spotbugs | 1m 32s | /branch-spotbugs-hbase-server-warnings.html | hbase-server in master has 1 extant spotbugs warnings. |
| +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 50s | the patch passed | |
| +1 :green_heart: | compile | 4m 5s | the patch passed | |
| +1 :green_heart: | cc | 4m 5s | the patch passed | |
| +1 :green_heart: | javac | 4m 5s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 38s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 4 new + 6 unchanged - 0 fixed = 10 total (was 6) |
| -1 :x: | spotbugs | 1m 41s | /new-spotbugs-hbase-server.html | hbase-server generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | hadoopcheck | 10m 31s | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. | |
| +1 :green_heart: | hbaseprotoc | 1m 33s | the patch passed | |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | |
| 46m 56s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hbase-server |
| Switch statement found in org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(MasterProcedureEnv, MasterProcedureProtos$DeleteTableState) where one case falls through to the next case At DeleteTableProcedure.java:MasterProcedureProtos$DeleteTableState) where one case falls through to the next case At DeleteTableProcedure.java:[lines 111-113] |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc |
| uname | Linux 1f6e2b98861b 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a6ad9e95ae1b30411488c89ae1a092fc2a0dd608 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 87 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 3m 2s | master passed | |
| +1 :green_heart: | compile | 1m 46s | master passed | |
| +1 :green_heart: | javadoc | 0m 48s | master passed | |
| +1 :green_heart: | shadedjars | 5m 30s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 52s | the patch passed | |
| +1 :green_heart: | compile | 1m 47s | the patch passed | |
| +1 :green_heart: | javac | 1m 47s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed | |
| +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 33s | hbase-protocol-shaded in the patch passed. | |
| -1 :x: | unit | 208m 54s | /patch-unit-hbase-server.txt | hbase-server in the patch failed. |
| +1 :green_heart: | unit | 0m 50s | hbase-it in the patch passed. | |
| 237m 28s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 7d8bf0be5ac9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a6ad9e95ae1b30411488c89ae1a092fc2a0dd608 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/1/testReport/ |
| Max. process+thread count | 5398 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
This failure I noticed locally overnight too...
[ERROR] org.apache.hadoop.hbase.master.procedure.TestTruncateRegionProcedure.testTruncateRegionProcedureErrorWhenSpecifiedReplicaRegionID [ERROR] Run 1: TestTruncateRegionProcedure.testTruncateRegionProcedureErrorWhenSpecifiedReplicaRegionID » IO java.lang.Exception: java.lang.AssertionError: Can't truncate replicas directly. Replicas are auto-truncated when their primary is truncated. [ERROR] Run 2: TestTruncateRegionProcedure.testTruncateRegionProcedureErrorWhenSpecifiedReplicaRegionID » IO java.lang.Exception: java.lang.AssertionError: Can't truncate replicas directly. Replicas are auto-truncated when their primary is truncated. [ERROR] Run 3: TestTruncateRegionProcedure.testTruncateRegionProcedureErrorWhenSpecifiedReplicaRegionID » IO java.lang.Exception: java.lang.AssertionError: Can't truncate replicas directly. Replicas are auto-truncated when their primary is truncated.
The changes to TruncateRegionProcedure are new in this patch as of yesterday when I ported our internal patch to this PR. We don't have this in branch-2.5 so don't have this change internally. Need to look at this.
The Spotbugs issue is arguable but I won't argue with it. It is my style to have switch statements where one case falls through to the next case deliberately.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 3m 21s | master passed | |
| +1 :green_heart: | compile | 4m 28s | master passed | |
| +1 :green_heart: | checkstyle | 1m 8s | master passed | |
| -1 :x: | spotbugs | 1m 46s | /branch-spotbugs-hbase-server-warnings.html | hbase-server in master has 1 extant spotbugs warnings. |
| +1 :green_heart: | spotless | 0m 49s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 1s | the patch passed | |
| +1 :green_heart: | compile | 4m 22s | the patch passed | |
| +1 :green_heart: | cc | 4m 22s | the patch passed | |
| +1 :green_heart: | javac | 4m 22s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 38s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 4 new + 6 unchanged - 0 fixed = 10 total (was 6) |
| -1 :x: | spotbugs | 1m 42s | /new-spotbugs-hbase-server.html | hbase-server generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | hadoopcheck | 10m 17s | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. | |
| +1 :green_heart: | hbaseprotoc | 1m 34s | the patch passed | |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | |
| 49m 24s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hbase-server |
| Switch statement found in org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(MasterProcedureEnv, MasterProcedureProtos$DeleteTableState) where one case falls through to the next case At DeleteTableProcedure.java:MasterProcedureProtos$DeleteTableState) where one case falls through to the next case At DeleteTableProcedure.java:[lines 111-113] |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc |
| uname | Linux a30855b664e0 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b2650065ef16aa8dff9e4141d38d972d769cebd0 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/2/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 2m 55s | master passed | |
| +1 :green_heart: | compile | 1m 48s | master passed | |
| +1 :green_heart: | javadoc | 0m 49s | master passed | |
| +1 :green_heart: | shadedjars | 5m 35s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 52s | the patch passed | |
| +1 :green_heart: | compile | 1m 50s | the patch passed | |
| +1 :green_heart: | javac | 1m 50s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed | |
| +1 :green_heart: | shadedjars | 5m 37s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 34s | hbase-protocol-shaded in the patch passed. | |
| +1 :green_heart: | unit | 215m 21s | hbase-server in the patch passed. | |
| +1 :green_heart: | unit | 0m 56s | hbase-it in the patch passed. | |
| 244m 25s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 47b9180c0e9c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b2650065ef16aa8dff9e4141d38d972d769cebd0 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/2/testReport/ |
| Max. process+thread count | 5065 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 3m 8s | master passed | |
| +1 :green_heart: | compile | 4m 18s | master passed | |
| +1 :green_heart: | checkstyle | 0m 54s | master passed | |
| -1 :x: | spotbugs | 1m 33s | /branch-spotbugs-hbase-server-warnings.html | hbase-server in master has 1 extant spotbugs warnings. |
| +1 :green_heart: | spotless | 0m 47s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 5s | the patch passed | |
| +1 :green_heart: | compile | 4m 7s | the patch passed | |
| +1 :green_heart: | cc | 4m 7s | the patch passed | |
| +1 :green_heart: | javac | 4m 7s | the patch passed | |
| -0 :warning: | blanks | 0m 0s | /blanks-tabs.txt | The patch 1 line(s) with tabs. |
| -0 :warning: | checkstyle | 0m 38s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 5 new + 6 unchanged - 0 fixed = 11 total (was 6) |
| +1 :green_heart: | spotbugs | 4m 42s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 11m 30s | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. | |
| +1 :green_heart: | hbaseprotoc | 1m 37s | the patch passed | |
| -1 :x: | spotless | 0m 41s | patch has 22 errors when running spotless:check, run spotless:apply to fix. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | |
| 49m 1s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc |
| uname | Linux a0e19c3a010e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a99c8a6bec566e3aaa03b57ed7aeb4f1fea65891 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/3/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/3/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 2m 49s | master passed | |
| +1 :green_heart: | compile | 1m 47s | master passed | |
| +1 :green_heart: | javadoc | 0m 48s | master passed | |
| +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 54s | the patch passed | |
| +1 :green_heart: | compile | 1m 47s | the patch passed | |
| +1 :green_heart: | javac | 1m 47s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed | |
| +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 35s | hbase-protocol-shaded in the patch passed. | |
| +1 :green_heart: | unit | 212m 10s | hbase-server in the patch passed. | |
| +1 :green_heart: | unit | 0m 49s | hbase-it in the patch passed. | |
| 240m 38s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 6c8424e89755 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a99c8a6bec566e3aaa03b57ed7aeb4f1fea65891 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/3/testReport/ |
| Max. process+thread count | 5167 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/3/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +0 :ok: | buf | 0m 0s | buf was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 2m 58s | master passed | |
| +1 :green_heart: | compile | 4m 3s | master passed | |
| +1 :green_heart: | checkstyle | 0m 51s | master passed | |
| -1 :x: | spotbugs | 1m 26s | /branch-spotbugs-hbase-server-warnings.html | hbase-server in master has 1 extant spotbugs warnings. |
| +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 57s | the patch passed | |
| +1 :green_heart: | compile | 4m 12s | the patch passed | |
| +1 :green_heart: | cc | 4m 12s | the patch passed | |
| +1 :green_heart: | javac | 4m 12s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 37s | /results-checkstyle-hbase-server.txt | hbase-server: The patch generated 4 new + 6 unchanged - 0 fixed = 10 total (was 6) |
| +1 :green_heart: | spotbugs | 4m 53s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 11m 35s | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. | |
| +1 :green_heart: | hbaseprotoc | 1m 33s | the patch passed | |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | |
| 48m 15s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc |
| uname | Linux 11701901e082 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a26f2837dffd491832f734208b27cd55aa286f59 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/4/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 2m 46s | master passed | |
| +1 :green_heart: | compile | 1m 46s | master passed | |
| +1 :green_heart: | javadoc | 0m 47s | master passed | |
| +1 :green_heart: | shadedjars | 5m 28s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 50s | the patch passed | |
| +1 :green_heart: | compile | 1m 47s | the patch passed | |
| +1 :green_heart: | javac | 1m 47s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 45s | the patch passed | |
| +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 33s | hbase-protocol-shaded in the patch passed. | |
| +1 :green_heart: | unit | 207m 58s | hbase-server in the patch passed. | |
| +1 :green_heart: | unit | 0m 47s | hbase-it in the patch passed. | |
| 236m 3s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6383 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux f202f1550aaa 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a26f2837dffd491832f734208b27cd55aa286f59 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/4/testReport/ |
| Max. process+thread count | 5872 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server hbase-it U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6383/4/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
I think I have answered questions and corrected misapprehensions, especially about blocking or non-blocking behavior in the procedure when the snapshot is executing (we are non-blocking, we suspend the procedure, giving the executor back, and get periodically rescheduled to continue once the snapshot is finished), and acknowledged the small improvements. Are we good to move forward with this? If so I will make the small changes that @virajjasani recommended. @Apache9 @shahrs87 @virajjasani
@Apache9
So why not just submit a child procedure for doing the snapshoting? In this way, after the sub procedure finishes, the parent procedure will continue again, so we do not need to add these timeout/check logic here.
Because the pattern to throw an exception to suspend the procedure executor with a timeout is already in the code, I think in the create table procedure if I remember correctly. Put there by you I believe :-) I used it as a template for this implementation.
In the design document the timeout is discussed as a design requirement. We do want the timeout.
The code in this PR as currently implemented is in production at my $dayjob and works well.
Edit: Even if implemented as a sub-procedure, although snapshots might use some procedures, it is mixed procedure code and non-procedure code. For disabled tables it is all executor work on the master. Even if we make a subprocedure to wait for the snapshot, we still tie up a procedure executor thread busy waiting on the snapshot code, unless we suspend/resume the would-be subprocedure the same way we currently suspend/resume the procedure directly. Adding a subprocedure doesn't help, it just makes things more complicated. Also because we targeted this change to go all the way back to 2.5 that impacted this decision too.
Thanks for the reviews @virajjasani @Apache9 . I will post an update soon.
Let me bump this. We are based on 2.5 where I work so the current PR takes an approach that can be different if we base this on 2.6 instead, which has HBASE-26323. Instead of exposing some internals of SnapshotManger we could run snapshots as a subprocedure. Some PR review comments touch on this. I plan to come back to this once we have moved up to 2.6.