spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard
verified publisher icon NVIDIA

Investigate CoalescedHashPartitioning

Open jlowe opened this issue 2 years ago • 3 comments

https://github.com/apache/spark/commit/b53c170679f added a new CoalescedHashPartitioning case class. We need to investigate whether the RAPIDS Accelerator needs to be updated to handle this new partitioning.

jlowe avatar Nov 28 '23 22:11 jlowe

See https://github.com/apache/spark/commit/81639090622 for changes that were needed to the CPU BroadcastHashJoinExec that are probably relevant to the changes likely needed for the GPU version.

jlowe avatar Nov 28 '23 22:11 jlowe

related to #9753

razajafri avatar Mar 11 '24 21:03 razajafri

See apache/spark@81639090622 for changes that were needed to the CPU BroadcastHashJoinExec that are probably relevant to the changes likely needed for the GPU version.

This commit updated the outputPartitioning logic in BroadcastHashJoinExec, but we do not currently implement this method in the GPU equivalent class. I filed https://github.com/NVIDIA/spark-rapids/issues/10609 to discuss whether we should also implement this, but as @revans2 pointed out, AQE will call the CPU version of this method anyway during replanning and before we replace these operators, so maybe we don't really need them.

andygrove avatar Mar 18 '24 22:03 andygrove