pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Replicate table with different config

Open xiangfu0 opened this issue 3 years ago • 1 comments

Use case: a table is bootstrapped already, however we want to test if how much perf boost partitioning a column will bring.

The only solution now is to create a new table, regenerate all the segments.

It would be better to have a way to check segment partition mechanism and repartition data if needed.

xiangfu0 avatar Sep 14 '22 00:09 xiangfu0

I would model it into 2 steps:

  1. Replicate a table
  2. Re-partition the data if necessary

The second step can be very expensive because it requires shuffle the data across segments, so we want to leverage minion to do that.

Jackie-Jiang avatar Sep 14 '22 21:09 Jackie-Jiang