Akka message(frame) size limit too small

[Yicong-Huang]

So there must be some limitation on the tuple size. Right now akka limit per message 120mb (we could config it higher but don’t know efficiency and stableness). Arrow limit per column 2GB (also experimental, recommended to be lower).

• For a short term, we can recommend users to use smaller tuples: since bigger tuple might affect performance.
• For the long term, I suggest we do three things:
  1. define a Texera tuple max size, and show an approaches error message to alert user over size tuple is found.
  2. Test and evaluate large message performance and stableness of akka and arrow. Depending on those evaluations, we decide the proper Texera tuple max size.
  3. Implement dynamic batch size.

[zuozhiw]

Discussion 2022.05.19: low priority

[Xiao-zhen-Liu]

Discussion 2022.12.07: Try increasing the limit to Akka limit and throw exception if limit is still exceeded. @shengquan-ni will take care of this one.

[shengquan-ni]

Setting the max frame size to 120MB is enough for now. We need to break down a large payload into smaller payloads and need another issue for the breakdown idea.