hbase icon indicating copy to clipboard operation
hbase copied to clipboard

HBASE-28627 REST ScannerModel doesn't support includeStartRow/include…

Open chandrasekhar-188k opened this issue 1 year ago • 2 comments

…StopRow

chandrasekhar-188k avatar Oct 20 '24 12:10 chandrasekhar-188k

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 2m 34s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 :ok: mvndep 0m 10s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 12s master passed
+1 :green_heart: compile 0m 51s master passed
+1 :green_heart: javadoc 0m 23s master passed
+1 :green_heart: shadedjars 5m 44s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 13s Maven dependency ordering for patch
+1 :green_heart: mvninstall 3m 6s the patch passed
+1 :green_heart: compile 0m 52s the patch passed
+1 :green_heart: javac 0m 52s the patch passed
+1 :green_heart: javadoc 0m 22s the patch passed
+1 :green_heart: shadedjars 5m 42s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 0m 33s hbase-protocol-shaded in the patch passed.
-1 :x: unit 0m 45s /patch-unit-hbase-rest.txt hbase-rest in the patch failed.
25m 36s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6374
Optional Tests unit javac javadoc compile shadedjars
uname Linux 66cd02a5214b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d6a342faa5eae250d47d4f41d952ad6addcb1751
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/1/testReport/
Max. process+thread count 221 (vs. ulimit of 30000)
modules C: hbase-protocol-shaded hbase-rest U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 20 '24 12:10 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 1m 1s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: buf 0m 0s buf was not available.
+0 :ok: buf 0m 0s buf was not available.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 :ok: mvndep 0m 10s Maven dependency ordering for branch
+1 :green_heart: mvninstall 3m 30s master passed
+1 :green_heart: compile 1m 25s master passed
+1 :green_heart: checkstyle 0m 19s master passed
+1 :green_heart: spotbugs 3m 15s master passed
+1 :green_heart: spotless 0m 48s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 :ok: mvndep 0m 11s Maven dependency ordering for patch
+1 :green_heart: mvninstall 2m 58s the patch passed
+1 :green_heart: compile 1m 14s the patch passed
+1 :green_heart: cc 1m 14s the patch passed
-0 :warning: javac 0m 34s /results-compile-javac-hbase-rest.txt hbase-rest generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183)
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 20s the patch passed
+1 :green_heart: spotbugs 2m 58s the patch passed
+1 :green_heart: hadoopcheck 10m 54s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 :green_heart: hbaseprotoc 1m 2s the patch passed
+1 :green_heart: spotless 0m 44s patch has no errors when running spotless:check.
_ Other Tests _
+1 :green_heart: asflicense 0m 21s The patch does not generate ASF License warnings.
38m 9s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6374
Optional Tests dupname asflicense cc buflint bufcompat codespell detsecrets hbaseprotoc spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti
uname Linux 66885f996707 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d6a342faa5eae250d47d4f41d952ad6addcb1751
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 86 (vs. ulimit of 30000)
modules C: hbase-protocol-shaded hbase-rest U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/1/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Oct 20 '24 12:10 Apache-HBase

Thank you.

The thorny issue here backwards-forward compatibility.

i.e. What happens if a client without this patch tries to run a scan on server with this change, and vice versa, with all three (json,xml,protobuf) encodings.

Unfortnuately, the easiest way to test that is bringing up an old/new cluster (can be a single process one), and trying a scan with the new/old client manually for all six cases.

To be clear, it's fine for the old server to fail when the new parameters are set, but it should work if they are not set, or are at the default value.

stoty avatar Oct 22 '24 14:10 stoty

Thank you.

The thorny issue here backwards-forward compatibility.

i.e. What happens if a client without this patch tries to run a scan on server with this change, and vice versa, with all three (json,xml,protobuf) encodings.

Unfortnuately, the easiest way to test that is bringing up an old/new cluster (can be a single process one), and trying a scan with the new/old client manually for all six cases.

To be clear, it's fine for the old server to fail when the new parameters are set, but it should work if they are not set, or are at the default value.

i have tested for JSON with new server and new/old clients, i will test for other combinations also and update the test results.

chandrasekhar-188k avatar Nov 07 '24 10:11 chandrasekhar-188k

i.e. What happens if a client without this patch tries to run a scan on server with this change, and vice versa, with all three (json,xml,protobuf) encodings.

I have tested for all the combinations and results are fine.

REST_Scan.ods image

chandrasekhar-188k avatar Nov 10 '24 01:11 chandrasekhar-188k

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 28s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 43s master passed
+1 :green_heart: compile 0m 20s master passed
+1 :green_heart: javadoc 0m 17s master passed
+1 :green_heart: shadedjars 5m 57s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 3s the patch passed
+1 :green_heart: compile 0m 20s the patch passed
+1 :green_heart: javac 0m 20s the patch passed
+1 :green_heart: javadoc 0m 16s the patch passed
+1 :green_heart: shadedjars 5m 51s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 :x: unit 0m 45s /patch-unit-hbase-rest.txt hbase-rest in the patch failed.
21m 59s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6374
Optional Tests javac javadoc unit compile shadedjars
uname Linux a0e03966ff1c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / a2e0e1ee89068452b726ef4f3442d8cd8eeaafe2
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/2/testReport/
Max. process+thread count 225 (vs. ulimit of 30000)
modules C: hbase-rest U: hbase-rest
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 10 '24 02:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 1m 4s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: buf 0m 0s buf was not available.
+0 :ok: buf 0m 0s buf was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 18s master passed
+1 :green_heart: compile 0m 37s master passed
+1 :green_heart: checkstyle 0m 12s master passed
+1 :green_heart: spotbugs 0m 40s master passed
+1 :green_heart: spotless 0m 44s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 2s the patch passed
+1 :green_heart: compile 0m 36s the patch passed
+1 :green_heart: cc 0m 36s the patch passed
-0 :warning: javac 0m 36s /results-compile-javac-hbase-rest.txt hbase-rest generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183)
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 12s the patch passed
+1 :green_heart: spotbugs 0m 43s the patch passed
+1 :green_heart: hadoopcheck 10m 46s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 :green_heart: hbaseprotoc 0m 22s the patch passed
+1 :green_heart: spotless 0m 44s patch has no errors when running spotless:check.
_ Other Tests _
+1 :green_heart: asflicense 0m 10s The patch does not generate ASF License warnings.
30m 16s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6374
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc
uname Linux 13ff4cbcfd58 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / a2e0e1ee89068452b726ef4f3442d8cd8eeaafe2
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 83 (vs. ulimit of 30000)
modules C: hbase-rest U: hbase-rest
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/2/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 10 '24 02:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 43s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+0 :ok: buf 0m 0s buf was not available.
+0 :ok: buf 0m 0s buf was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 16s master passed
+1 :green_heart: compile 0m 45s master passed
+1 :green_heart: checkstyle 0m 16s master passed
+1 :green_heart: spotbugs 0m 57s master passed
+1 :green_heart: spotless 1m 5s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 34s the patch passed
+1 :green_heart: compile 0m 43s the patch passed
+1 :green_heart: cc 0m 43s the patch passed
-0 :warning: javac 0m 43s /results-compile-javac-hbase-rest.txt hbase-rest generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183)
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 12s the patch passed
+1 :green_heart: spotbugs 0m 54s the patch passed
+1 :green_heart: hadoopcheck 12m 22s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 :green_heart: hbaseprotoc 0m 26s the patch passed
+1 :green_heart: spotless 0m 54s patch has no errors when running spotless:check.
_ Other Tests _
+1 :green_heart: asflicense 0m 11s The patch does not generate ASF License warnings.
33m 58s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6374
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc
uname Linux 4c1f57507408 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 2d3a36d841aaf3037a056304e1a49211abef7ebe
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 84 (vs. ulimit of 30000)
modules C: hbase-rest U: hbase-rest
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/3/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 11 '24 13:11 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 41s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 2s master passed
+1 :green_heart: compile 0m 23s master passed
+1 :green_heart: javadoc 0m 19s master passed
+1 :green_heart: shadedjars 5m 22s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 52s the patch passed
+1 :green_heart: compile 0m 24s the patch passed
+1 :green_heart: javac 0m 24s the patch passed
+1 :green_heart: javadoc 0m 18s the patch passed
+1 :green_heart: shadedjars 5m 13s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 :green_heart: unit 3m 39s hbase-rest in the patch passed.
23m 18s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/6374
Optional Tests javac javadoc unit compile shadedjars
uname Linux 9ea8555c21e5 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 2d3a36d841aaf3037a056304e1a49211abef7ebe
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/3/testReport/
Max. process+thread count 1542 (vs. ulimit of 30000)
modules C: hbase-rest U: hbase-rest
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6374/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Nov 11 '24 13:11 Apache-HBase

Thanks.

Can you please do a backport PR for branch-2 @chandrasekhar-188k ?

stoty avatar Nov 25 '24 09:11 stoty

Thanks.

Can you please do a backport PR for branch-2 @chandrasekhar-188k ?

okay

chandrasekhar-188k avatar Nov 26 '24 14:11 chandrasekhar-188k

Thanks.

Can you please do a backport PR for branch-2 @chandrasekhar-188k ?

backport PR raised for branch-2, please review https://github.com/apache/hbase/pull/6494

chandrasekhar-188k avatar Nov 26 '24 14:11 chandrasekhar-188k

After commiting this (a little late), I realized that I have more questions:

I see this annotation on ScannerModel:

@JsonInclude(JsonInclude.Include.NON_NULL)

and

private boolean includeStartRow = true;

private boolean includeStopRow = false;

Doesn't that mean that the new attributes will get serialized regardless of their values to JSON, and cause the deserialization in the old server to fail ?

You matrix shows this as passing, but I can't see how that can happen. Does JSON skip serializing the default values ?

stoty avatar Nov 27 '24 10:11 stoty

After commiting this (a little late), I realized that I have more questions:

I see this annotation on ScannerModel:

@JsonInclude(JsonInclude.Include.NON_NULL)

and

private boolean includeStartRow = true; private boolean includeStopRow = false;

Doesn't that mean that the new attributes will get serialized regardless of their values to JSON, and cause the deserialization in the old server to fail ?

You matrix shows this as passing, but I can't see how that can happen. Does JSON skip serializing the default values ?

you are correct, when we serialize the ScannerModel the new attributes are also included and causing the deserilization to fail at old server. when I tested for JSON I have construcuted the JSON message manually with the required fields, instead of Converting the ScannerModel to JSON, that's why my test was ok.
For XML and protobuff I had tested with Conversion of ScannerModel object to the required mime type, so no problem here.

For solving the issue wrt JSON, I think we can have two approaches

  1. change the class level JSON include annotation to @JsonInclude(JsonInclude.Include.NON_DEFAULT), so that it doesn't serialize the default values. This can work with out any functional impact only if the default values of ScannerModel and Scan Class are in sync always.. but currently I could see the default values for some fields are different between ScannerModel and Scan.
  2. For the newly introduced attributes add a custom JSON include annotation with a filter class that skips serialization of a field if the field has default value set to it. ex:
@JsonInclude(value = JsonInclude.Include.CUSTOM, valueFilter = MaskingFilter.class)
private boolean includeStartRow = true;

static class MaskingFilter {
   public boolean equals(Object value){
     return Boolean.TRUE.equals(value);
}

}

I think the 2nd approach is better for this. I will raise an addendum PR for this..Let me know if you have any other suggestions..

chandrasekhar-188k avatar Nov 27 '24 18:11 chandrasekhar-188k

After commiting this (a little late), I realized that I have more questions: I see this annotation on ScannerModel:

@JsonInclude(JsonInclude.Include.NON_NULL)

and

private boolean includeStartRow = true; private boolean includeStopRow = false;

Doesn't that mean that the new attributes will get serialized regardless of their values to JSON, and cause the deserialization in the old server to fail ? You matrix shows this as passing, but I can't see how that can happen. Does JSON skip serializing the default values ?

you are correct, when we serialize the ScannerModel the new attributes are also included and causing the deserilization to fail at old server. when I tested for JSON I have construcuted the JSON message manually with the required fields, instead of Converting the ScannerModel to JSON, that's why my test was ok. For XML and protobuff I had tested with Conversion of ScannerModel object to the required mime type, so no problem here.

For solving the issue wrt JSON, I think we can have two approaches

  1. change the class level JSON include annotation to @JsonInclude(JsonInclude.Include.NON_DEFAULT), so that it doesn't serialize the default values. This can work with out any functional impact only if the default values of ScannerModel and Scan Class are in sync always.. but currently I could see the default values for some fields are different between ScannerModel and Scan.
  2. For the newly introduced attributes add a custom JSON include annotation with a filter class that skips serialization of a field if the field has default value set to it. ex:
@JsonInclude(value = JsonInclude.Include.CUSTOM, valueFilter = MaskingFilter.class)
private boolean includeStartRow = true;

static class MaskingFilter {
   public boolean equals(Object value){
     return Boolean.TRUE.equals(value);
}

}

I think the 2nd approach is better for this. I will raise an addendum PR for this..Let me know if you have any other suggestions..

addendum PR raised for this fix, Please review https://github.com/apache/hbase/pull/6499

chandrasekhar-188k avatar Nov 28 '24 16:11 chandrasekhar-188k

Can you fix the test and answer my question question by mid next week @chandrasekhar-188k ?

I don't want to start the winter holidays with the unfixed compatibility issue, so if we cannot fix that, I will temporarily revert the patch until the compatibility issue can be fixed.

stoty avatar Dec 06 '24 07:12 stoty