spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Address Misc issues after merging AutoTuner feature

Open amahussein opened this issue 2 years ago • 0 comments

Followup on #6301 and the ~~PR #6301~~ PR #6338

Some remaining comments in the PR that were not addressed yet:

  • https://github.com/NVIDIA/spark-rapids/pull/6301/files/88149ac1084f2e8ee0f8650e455d3a491204dede#r944474110 make recommendation for 'spark.sql.shuffle.partitions' based on spill size optional recommendation since we are working on heuristics

  • https://github.com/NVIDIA/spark-rapids/pull/6301/files/88149ac1084f2e8ee0f8650e455d3a491204dede#r944479548
    there is a lot of converting from human readable in here, if we create these global vars it might be better to use the non-human readable until we go to actually print something to the user, or have getters that do it

  • https://github.com/NVIDIA/spark-rapids/pull/6301#discussion_r944466377 document what MIN_MEMORY_OVERHEAD is

amahussein avatar Aug 15 '22 15:08 amahussein