datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Fall back to Spark if query uses DPP to avoid perf regressions in TPC-DS

Open andygrove opened this issue 1 year ago • 1 comments

What is the problem the feature request solves?

Comet does not yet support DPP and this can result in poor performance on the TPC-DS benchmark due to scanning more Parquet files than needed.

Describe the potential solution

We should implement a rule to fall back to Spark for plans that use DPP to avoid major performance regressions.

Additional context

No response

andygrove avatar Aug 30 '24 16:08 andygrove

This is resolved for v1 data sources but not for v2.

andygrove avatar Sep 20 '24 16:09 andygrove