datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Misleading withInfo call for `AQEShuffleRead is not supported`

Open andygrove opened this issue 1 year ago • 0 comments

Describe the bug

We do not need to have a native version of AQEShuffleRead, so we should not show warnings like this:

WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
CometHashAggregate
+- CometSinkPlaceHolder
   +- CometExchange
      +- CometHashAggregate
         +- CometProject
            +- CometSortMergeJoin
               :- CometSort
               :  +- CometHashAggregate
               :     +-  AQEShuffleRead [COMET: AQEShuffleRead is not supported]
               :        +- CometExchange
               :           +- CometHashAggregate
               :              +- CometProject
               :                 +- CometBroadcastHashJoin
               :                    :- CometFilter
               :                    :  +- CometScan parquet 
               :                    +- CometBroadcastExchange
               :                       +- CometProject
               :                          +- CometFilter
               :                             +- CometScan parquet 
               +- CometSort
                  +- CometHashAggregate
                     +-  AQEShuffleRead [COMET: AQEShuffleRead is not supported]
                        +- CometExchange
                           +- CometHashAggregate
                              +- CometProject
                                 +- CometBroadcastHashJoin
                                    :- CometFilter
                                    :  +- CometScan parquet 
                                    +- ReusedExchange

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove avatar Aug 06 '24 13:08 andygrove