hail icon indicating copy to clipboard operation
hail copied to clipboard

[compiler] Redesign Repartition IR nodes to be naive coalesce only

Open tpoterba opened this issue 1 year ago • 0 comments

This changes the execution semantics for the Spark backend. Instead of dispatching either to (a) a two-pass algorithm that scans and coalesces (shuffle=False) or (b) a two-pass algorithm that shuffles and then rekeys (shuffle=True), we use write/read instead.

tpoterba avatar Feb 08 '23 15:02 tpoterba