hail
hail copied to clipboard
[compiler] Redesign Repartition IR nodes to be naive coalesce only
This changes the execution semantics for the Spark backend. Instead of dispatching either to (a) a two-pass algorithm that scans and coalesces (shuffle=False) or (b) a two-pass algorithm that shuffles and then rekeys (shuffle=True), we use write/read instead.