datafusion-comet
datafusion-comet copied to clipboard
Fall back to Spark if query uses DPP to avoid perf regressions in TPC-DS
What is the problem the feature request solves?
Comet does not yet support DPP and this can result in poor performance on the TPC-DS benchmark due to scanning more Parquet files than needed.
Describe the potential solution
We should implement a rule to fall back to Spark for plans that use DPP to avoid major performance regressions.
Additional context
No response
This is resolved for v1 data sources but not for v2.