pinot
pinot copied to clipboard
[multistage] split data table when size is too large
Problem
Currently we do not split data table when size is too large. thus we set the default GRPC channel msg size to 128MB. however this is still an issue when join two large tables together (especially using broadcast JOIN)
Propose
- introduce data table splitter so that each data block is small enough. this also help pipeline processing and speed up the overall latency.
- allow mailbox to send data blocks with variating number of splits.