pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Added API to check if 2 tables are co-located

Open suyashpatel98 opened this issue 1 year ago • 2 comments

Addresses #11788.

Approach: For each partition get the set of servers on which the partition resides for each table and verify that the 2 server sets are equal

Edge case: Consider table0 (with replication factor: 2) and table1 (replication factor: 1) server0: segment0_table0_replica_0, segment0_table1_replica_0 server1: segment0_table0_replia_1

In the above case the API will return false i.e tables aren't co-located (because the server sets must match exactly).

Notes: The reason I am returning a string response ("true" or "false") is so that we can decide in the UI how to display it to the user

Testing: Tested both positive and negative scenarios

testing-colocated-api testing-negative

suyashpatel98 avatar Jun 16 '24 04:06 suyashpatel98

Codecov Report

Attention: Patch coverage is 0% with 25 lines in your changes missing coverage. Please review.

Project coverage is 61.84%. Comparing base (59551e4) to head (ccbd8be). Report is 632 commits behind head on master.

Files Patch % Lines
...oller/api/resources/PinotTableRestletResource.java 0.00% 25 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #13403      +/-   ##
============================================
+ Coverage     61.75%   61.84%   +0.09%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2550     +114     
  Lines        133233   140311    +7078     
  Branches      20636    21812    +1176     
============================================
+ Hits          82274    86777    +4503     
- Misses        44911    46924    +2013     
- Partials       6048     6610     +562     
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (-0.01%) :arrow_down:
integration <0.01% <0.00%> (-0.01%) :arrow_down:
integration1 <0.01% <0.00%> (-0.01%) :arrow_down:
integration2 0.00% <0.00%> (ø)
java-11 27.65% <0.00%> (-34.06%) :arrow_down:
java-21 61.84% <0.00%> (+0.21%) :arrow_up:
skip-bytebuffers-false 61.82% <0.00%> (+0.07%) :arrow_up:
skip-bytebuffers-true 61.82% <0.00%> (+34.10%) :arrow_up:
temurin 61.84% <0.00%> (+0.09%) :arrow_up:
unittests 61.84% <0.00%> (+0.09%) :arrow_up:
unittests1 46.38% <ø> (-0.51%) :arrow_down:
unittests2 27.65% <0.00%> (-0.08%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Jun 16 '24 04:06 codecov-commenter

On second thought these is something wrong with the implementation - we shouldn't be getting segment names and comparing them. My implementation works locally because

IdealState table2IdealState = _pinotHelixResourceManager.getHelixAdmin()
.getResourceIdealState(_pinotHelixResourceManager.getHelixClusterName(), table2);

table1IdealState.getPartitionSet() contains eventData_OFFLINE_1717648729000_1717883732000_0 and table2IdealState.getPartitionSet() contains eventData_OFFLINE_1717648729000_1717883732000_0, transcript_OFFLINE_1570863600000_1572418800000_0

for some reason table2IdealState.getPartitionSet() contains eventData_OFFLINE_1717648729000_1717883732000_0 and transcript_OFFLINE_1570863600000_1572418800000_0 which I think is a bug.

@xiangfu0 @Jackie-Jiang Do you think table2IdealState.getPartitionSet() returning a segment from another table is a bug? Also, what approach would you suggest to implement this? Thanks!

suyashpatel98 avatar Jun 16 '24 05:06 suyashpatel98