blaze
blaze copied to clipboard
`spark.blaze.enable.shuffle` doesn't work
Describe the bug
I try to disable coverting ShuffleExchangeExec
, but it doesn't work. And i found the code
def convertSparkPlan(exec: SparkPlan): SparkPlan = {
exec match {
case e: ShuffleExchangeExec => tryConvert(e, convertShuffleExchangeExec)
.....
It's not dependent on the configuration spark.blaze.enable.shuffle
.
To Reproduce
Add configuration --conf spark.blaze.enable.shuffle=false
when submit
Expected behavior
Blaze extension should not covert ShuffleExchangeExec
if spark.blaze.enable.shuffle
is true.
Additional context
ping @richox
ping @yjshen