hbase icon indicating copy to clipboard operation
hbase copied to clipboard

HBASE-29716 Include sequence ID metadata for HFiles generated by incremental backups

Open krconv opened this issue 1 month ago • 8 comments

The HFiles generated by incremental backups cannot be properly read by tooling such as the ClientSideRequestScanner, because the generated HFiles do not include the MAX_SEQ_ID metadata. The scanner will ignore cell-level sequence IDs and instead sort the HFiles arbitrarily. This causes incorrect results when scanning overwrites to cells with the same timestamp.

This PR adds a new option to the HFileOutputFormat2 that will calculate and set the required metadata. This only really effects the ClientSideRequestScanner, as the sequence ID will be recalculated when bulk-loaded anyways.

Part of https://issues.apache.org/jira/browse/HBASE-29716

krconv avatar Nov 24 '25 23:11 krconv

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 :ok: mvndep 0m 17s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 34s master passed
+1 :green_heart: compile 1m 9s master passed
+1 :green_heart: checkstyle 0m 26s master passed
+1 :green_heart: spotbugs 1m 5s master passed
+1 :green_heart: spotless 0m 53s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 7s the patch passed
+1 :green_heart: compile 1m 7s the patch passed
+1 :green_heart: javac 1m 7s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 24s the patch passed
+1 :green_heart: spotbugs 1m 14s the patch passed
+1 :green_heart: hadoopcheck 12m 19s Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 :green_heart: spotless 0m 46s patch has no errors when running spotless:check.
_ Other Tests _
+1 :green_heart: asflicense 0m 18s The patch does not generate ASF License warnings.
35m 43s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 7b53662b9750 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 7317c4d2d011f9596eeb87b83723dc0f08c86a70
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 84 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/1/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 25 '25 00:11 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 32s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 :ok: mvndep 0m 16s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 31s master passed
+1 :green_heart: compile 0m 39s master passed
+1 :green_heart: javadoc 0m 29s master passed
+1 :green_heart: shadedjars 6m 14s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 13s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 5s the patch passed
+1 :green_heart: compile 0m 39s the patch passed
+1 :green_heart: javac 0m 39s the patch passed
+1 :green_heart: javadoc 0m 26s the patch passed
+1 :green_heart: shadedjars 6m 6s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 :x: unit 17m 34s /patch-unit-hbase-mapreduce.txt hbase-mapreduce in the patch failed.
+1 :green_heart: unit 10m 8s hbase-backup in the patch passed.
51m 14s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests javac javadoc unit compile shadedjars
uname Linux 960fe9326030 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 7317c4d2d011f9596eeb87b83723dc0f08c86a70
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/1/testReport/
Max. process+thread count 3308 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 25 '25 00:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 :ok: mvndep 0m 15s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 32s master passed
+1 :green_heart: compile 1m 9s master passed
+1 :green_heart: checkstyle 0m 26s master passed
+1 :green_heart: spotbugs 1m 3s master passed
+1 :green_heart: spotless 0m 54s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 11s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 9s the patch passed
+1 :green_heart: compile 1m 7s the patch passed
+1 :green_heart: javac 1m 7s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 25s the patch passed
+1 :green_heart: spotbugs 1m 12s the patch passed
+1 :green_heart: hadoopcheck 12m 21s Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 :green_heart: spotless 0m 46s patch has no errors when running spotless:check.
_ Other Tests _
+1 :green_heart: asflicense 0m 18s The patch does not generate ASF License warnings.
35m 20s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 686f24cd0cdf 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 45234b1ef12743d0e83125464440d83818eaf5a0
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 84 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/2/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 25 '25 02:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 28s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 5s master passed
+1 :green_heart: compile 0m 38s master passed
+1 :green_heart: javadoc 0m 26s master passed
+1 :green_heart: shadedjars 6m 7s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 13s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 5s the patch passed
+1 :green_heart: compile 0m 39s the patch passed
+1 :green_heart: javac 0m 39s the patch passed
+1 :green_heart: javadoc 0m 27s the patch passed
+1 :green_heart: shadedjars 6m 7s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 17m 21s hbase-mapreduce in the patch passed.
+1 :green_heart: unit 10m 9s hbase-backup in the patch passed.
50m 18s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests javac javadoc unit compile shadedjars
uname Linux d47e91a726ba 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 45234b1ef12743d0e83125464440d83818eaf5a0
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/2/testReport/
Max. process+thread count 3301 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 25 '25 03:11 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 8s master passed
+1 :green_heart: compile 1m 7s master passed
+1 :green_heart: checkstyle 0m 24s master passed
+1 :green_heart: spotbugs 0m 59s master passed
+1 :green_heart: spotless 0m 46s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 12s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 7s the patch passed
+1 :green_heart: compile 1m 6s the patch passed
+1 :green_heart: javac 1m 6s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 24s the patch passed
+1 :green_heart: spotbugs 1m 13s the patch passed
+1 :green_heart: hadoopcheck 12m 3s Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
-1 :x: spotless 0m 41s patch has 21 errors when running spotless:check, run spotless:apply to fix.
_ Other Tests _
+1 :green_heart: asflicense 0m 17s The patch does not generate ASF License warnings.
34m 5s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 40d18c1db355 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d4409697803f5c06073e9bb7acc117cef4845947
Default Java Eclipse Adoptium-17.0.11+9
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/3/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 85 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/3/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 25 '25 19:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 51s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 :ok: mvndep 0m 18s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 39s master passed
+1 :green_heart: compile 0m 40s master passed
+1 :green_heart: javadoc 0m 30s master passed
+1 :green_heart: shadedjars 6m 19s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 14s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 11s the patch passed
+1 :green_heart: compile 0m 40s the patch passed
+1 :green_heart: javac 0m 40s the patch passed
+1 :green_heart: javadoc 0m 27s the patch passed
+1 :green_heart: shadedjars 6m 13s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 17m 43s hbase-mapreduce in the patch passed.
+1 :green_heart: unit 10m 27s hbase-backup in the patch passed.
52m 31s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests javac javadoc unit compile shadedjars
uname Linux 9d35dab9ffa4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d4409697803f5c06073e9bb7acc117cef4845947
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/3/testReport/
Max. process+thread count 3260 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 25 '25 19:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 :ok: mvndep 0m 34s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 32s master passed
+1 :green_heart: compile 1m 12s master passed
+1 :green_heart: checkstyle 0m 28s master passed
+1 :green_heart: spotbugs 1m 3s master passed
+1 :green_heart: spotless 0m 49s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 13s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 12s the patch passed
+1 :green_heart: compile 1m 7s the patch passed
+1 :green_heart: javac 1m 7s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 25s the patch passed
+1 :green_heart: spotbugs 1m 13s the patch passed
+1 :green_heart: hadoopcheck 12m 15s Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 :green_heart: spotless 0m 46s patch has no errors when running spotless:check.
_ Other Tests _
+1 :green_heart: asflicense 0m 17s The patch does not generate ASF License warnings.
35m 40s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 6707fe2a3565 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e4921a102933b8930e2225d4d4163f9ed0b36337
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 83 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/4/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Dec 02 '25 13:12 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 29s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 :ok: mvndep 0m 33s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 30s master passed
+1 :green_heart: compile 0m 41s master passed
+1 :green_heart: javadoc 0m 28s master passed
+1 :green_heart: shadedjars 6m 15s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 14s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 8s the patch passed
+1 :green_heart: compile 0m 40s the patch passed
+1 :green_heart: javac 0m 40s the patch passed
+1 :green_heart: javadoc 0m 27s the patch passed
+1 :green_heart: shadedjars 6m 10s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 17m 43s hbase-mapreduce in the patch passed.
+1 :green_heart: unit 10m 16s hbase-backup in the patch passed.
52m 6s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/7480
Optional Tests javac javadoc unit compile shadedjars
uname Linux 2df0d15dc25e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e4921a102933b8930e2225d4d4163f9ed0b36337
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/4/testReport/
Max. process+thread count 3131 (vs. ulimit of 30000)
modules C: hbase-mapreduce hbase-backup U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7480/4/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Dec 02 '25 13:12 Apache-HBase