[Upsert] persist validDocsIndex snapshot for Pinot upsert optimization

[deemoliu]

Part of a series of PRs for Pinot Upsert Optimizations.

Check this doc out for Upsert TTL and Snapshot Design

Heap memory usage is one of the pain point for Pinot upsert. Upsert TTL support will help use cases with live session (each records only alive for a short period of time, callled TTL). With upsert TTL support, out-of-date primary keys can be removed from heap and numbers of active primary keys stored in heap will be reduced.

TTL support will brings challenge when recovering validDocsIndexes after restart, because validDocIndexes requires to have full history of metadata data. After out-of-dated primarys get removed, validDocIndexes can not be recovered easily.

This PR address the challenging requirement by persisting and loading validDocSnapshots to disk to recover validDocIndex.

[deemoliu]

Will fix the checkstyle, test failures, and add unit tests

[codecov-commenter]

Codecov Report

Merging #9062 (5f77a15) into master (3818397) will increase coverage by 35.27%. The diff coverage is 39.70%.

@@              Coverage Diff              @@
##             master    #9062       +/-   ##
=============================================
+ Coverage     34.73%   70.00%   +35.27%     
- Complexity      194     5205     +5011     
=============================================
  Files          1910     1924       +14     
  Lines        101850   102470      +620     
  Branches      15452    15547       +95     
=============================================
+ Hits          35379    71739    +36360     
+ Misses        63483    25667    -37816     
- Partials       2988     5064     +2076     

Flag	Coverage Δ
integration1	26.00% <0.00%> (?)
integration2	24.71% <0.00%> (-0.02%)	:arrow_down:
unittests1	67.27% <39.70%> (?)
unittests2	15.57% <0.00%> (+0.08%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...t/ConcurrentMapPartitionUpsertMetadataManager.java	69.90% <ø> (+69.90%)	:arrow_up:
...psert/ConcurrentMapTableUpsertMetadataManager.java	50.00% <ø> (+50.00%)	:arrow_up:
...apache/pinot/segment/local/upsert/UpsertUtils.java	24.00% <0.00%> (+24.00%)	:arrow_up:
...java/org/apache/pinot/segment/spi/V1Constants.java	12.50% <ø> (+12.50%)	:arrow_up:
...cal/upsert/BasePartitionUpsertMetadataManager.java	55.46% <19.04%> (+55.46%)	:arrow_up:
...l/indexsegment/immutable/ImmutableSegmentImpl.java	60.00% <63.33%> (+60.00%)	:arrow_up:
...t/local/upsert/BaseTableUpsertMetadataManager.java	57.89% <100.00%> (+57.89%)	:arrow_up:
...rg/apache/pinot/spi/config/table/UpsertConfig.java	75.00% <100.00%> (+75.00%)	:arrow_up:
...java/org/apache/pinot/query/planner/QueryPlan.java	22.85% <0.00%> (-77.15%)	:arrow_down:
...lugin/stream/kafka20/server/KafkaDataProducer.java	33.33% <0.00%> (-24.36%)	:arrow_down:
... and 1184 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[KKcorps]

Is there a reason we can't enable this by default and need a config?

[deemoliu]

Is there a reason we can't enable this by default and need a config?

Hi @KKcorps thanks for reviewing. I think snapshotEnabled can be used as a config field similiar to "nullHandlingEnabled". It will not affect the backward compatibility. Also i am think about refactor this config once the TTL config is added, so it will be by default enabled for all the upsert TTL use cases (since it's required).

What do you think?

[deemoliu]

@Jackie-Jiang pointed that there is a potential issue cause inconsistent query result. #9095 #9101

when reloading the segments, the invalidDocsIndexes can be empty (drop and recreate) for a short period of time. If the persisting snapshot triggered during that time, we will persist empty snapshot and get wrong data. will need to rebase the change and then walk through corner cases when persist the snapshot.

[deemoliu]

@yupeng9 @Jackie-Jiang Do you have any suggestion on when to persist snapshot? I looked into partitionUpsertMetaDataManager and summarized the scenarios for persisting snapshot

[navina]

@yupeng9 / @Jackie-Jiang / @deemoliu can you please provide access to the design doc - this doc ?