spark-rapids
spark-rapids copied to clipboard
parameterize threshold for growth of single pass partial sort agg
add a config called spark.rapids.sql.agg.singlePassPartialSort.estimatedGrowthThreshold. (default value 1.1)
Even if spark.rapids.sql.agg.singlePassPartialSortEnabled is true, if the estimated growth of size per row is less than this config, single pass partial sort will not be enabled.
This PR has no related issue yet, we'll evaluate the necessarity first at customer side