pinot icon indicating copy to clipboard operation
pinot copied to clipboard

[multistage] split data table when size is too large

Open walterddr opened this issue 3 years ago • 0 comments

Problem

Currently we do not split data table when size is too large. thus we set the default GRPC channel msg size to 128MB. however this is still an issue when join two large tables together (especially using broadcast JOIN)

Propose

  • introduce data table splitter so that each data block is small enough. this also help pipeline processing and speed up the overall latency.
  • allow mailbox to send data blocks with variating number of splits.

walterddr avatar Aug 23 '22 23:08 walterddr