pinot icon indicating copy to clipboard operation
pinot copied to clipboard

PageCache warmup on Server Restarts and Segment Refresh

Open praveenc7 opened this issue 7 months ago • 1 comments

Summary

Issue link : https://github.com/apache/pinot/issues/15675

🚀 What this PR delivers

Screenshot 2025-06-22 at 3 32 48 PM

Screenshot 2025-06-19 at 11 31 50 AM


🔄 Typical flow

  1. Create Query file

curl -X POST /pagecache/queries/airlineStats?tableType=OFFLINE \
     -d '["SELECT …", "SELECT …"]' -H 'Content-Type: application/json'

2. Server restart

Startup hook calls warm-up for each enabled table; queries run for the configured warmup duration

3. Segment refresh

Controller triggers warm-up for only the newly uploaded segments using the consistent protocol feature.


Backward compatibility

  • Fully opt-in – no existing table is affected until pageCacheWarmupConfig is set.

  • Query path remains unchanged; warm-up traffic stays on the secondary queue during refresh if BinaryWorkloadScheduler is enabled. During restart it uses the same primary queue since it is not competing with live traffic

Testing

unit test, manually tested using quick-starter and test actual servers

praveenc7 avatar Jun 06 '25 19:06 praveenc7

Codecov Report

:x: Patch coverage is 23.51695% with 361 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 63.13%. Comparing base (62b20d9) to head (b19b478). :warning: Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
.../api/resources/PageCacheWarmupRestletResource.java 0.00% 94 Missing :warning:
...roller/util/PageCacheWarmupControllerExecutor.java 13.68% 79 Missing and 3 partials :warning:
...ver/warmup/PageCacheWarmupServerQueryExecutor.java 54.42% 51 Missing and 16 partials :warning:
...pinot/server/warmup/PageCacheWarmupQueryUtils.java 0.00% 37 Missing :warning:
...che/pinot/server/api/resources/TablesResource.java 3.84% 25 Missing :warning:
...inot/controller/helix/ControllerRequestClient.java 0.00% 17 Missing :warning:
.../pinot/spi/config/table/PageCacheWarmupConfig.java 0.00% 16 Missing :warning:
.../pinot/server/starter/helix/BaseServerStarter.java 0.00% 7 Missing :warning:
...pinot/spi/config/table/PageCacheWarmupRequest.java 0.00% 6 Missing :warning:
...rg/apache/pinot/server/starter/ServerInstance.java 0.00% 3 Missing :warning:
... and 4 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #16033      +/-   ##
============================================
- Coverage     63.21%   63.13%   -0.09%     
+ Complexity     1428     1422       -6     
============================================
  Files          3120     3126       +6     
  Lines        184759   185228     +469     
  Branches      28320    28368      +48     
============================================
+ Hits         116801   116947     +146     
- Misses        58935    59246     +311     
- Partials       9023     9035      +12     
Flag Coverage Δ
custom-integration1 ?
integration ?
integration1 ?
integration2 ?
java-11 63.11% <23.51%> (-0.07%) :arrow_down:
java-21 63.07% <23.51%> (-0.11%) :arrow_down:
temurin 63.13% <23.51%> (-0.09%) :arrow_down:
unittests 63.13% <23.51%> (-0.08%) :arrow_down:
unittests1 55.96% <29.26%> (+0.02%) :arrow_up:
unittests2 33.73% <23.51%> (+<0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov-commenter avatar Nov 16 '25 23:11 codecov-commenter