Compressed Shuffle (Arrow-IPC compression)

Open yjshen opened this issue 4 years ago • 4 comments

Upstream issues: [TODO] Rust side: https://github.com/apache/arrow-rs/issues/1709 [Partly Finished?] Java side: https://issues.apache.org/jira/browse/ARROW-8672

Jan 11 '22 11:01 yjshen

IPC block-based compression is supported now. we can still switch to column-based compression if it achieves better compression and performance.

Jun 02 '22 07:06 richox

Great work! Is that possible to report new benchmark results for the latest master? @richox

Jun 02 '22 08:06 yjshen

We could always explore buffer based compression when it gets direct support from arrow-rs later.

Jun 02 '22 08:06 yjshen

Great work! Is that possible to report new benchmark results for the latest master? @richox

we got some performance issue when running on STS with small memory and broadcast join enabled. i guest we have to implement native BHJ before we get a better benchmark result.

Jun 08 '22 02:06 richox