datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Fallback reason missing for incompatible casts

Open andygrove opened this issue 10 months ago • 0 comments

Describe the bug

I see a HashAggregate falling back to Spark but the root cause is hidden:

HashAggregate [COMET: Unsupported result expressions found in: List((0.2 * cast((avg(UnscaledValue(l_quantity#130))#124 / 100.0) as decimal(15,6))) AS (0.2 * avg(l_quantity))#125, l_partkey#127L)]

After adding some debug logging I discover the root cause:

Comet does not guarantee correct results for cast from DoubleType to DecimalType(15,6) with timezone Some(America/Denver) and evalMode LEGACY (There can be rounding differences). To enable all incompatible casts, set spark.comet.cast.allowIncompatible=true

I should not need to add debug logging to discover this.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove avatar Feb 20 '25 14:02 andygrove