spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

parameterize threshold for growth of single pass partial sort agg

Open binmahone opened this issue 7 months ago • 1 comments

add a config called spark.rapids.sql.agg.singlePassPartialSort.estimatedGrowthThreshold. (default value 1.1)

Even if spark.rapids.sql.agg.singlePassPartialSortEnabled is true, if the estimated growth of size per row is less than this config, single pass partial sort will not be enabled.

This PR has no related issue yet, we'll evaluate the necessarity first at customer side

binmahone avatar Jul 02 '24 06:07 binmahone