spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

repartition-based fallback for hash aggregate

Open binmahone opened this issue 7 months ago • 18 comments

this PR closes https://github.com/NVIDIA/spark-rapids/issues/8391.

this PR add a config called spark.rapids.sql.agg.fallbackAlgorithm to let user decide a sort-based algorithm or repartition-based algorithm to use when agg cannot be done in a single pass in memory.

This optimization is orthogonal to https://github.com/NVIDIA/spark-rapids/pull/10950

binmahone avatar Jul 01 '24 09:07 binmahone