datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Spark SQL test failures due to timestamp mismatch when LocalTableScan is native

Open andygrove opened this issue 1 month ago • 1 comments

Describe the bug

I saw this in Spark SQL tests with spark.comet.sparkToColumnar.enabled=true.

2025-11-06T19:59:48.4097661Z [info] - sorting on TimestampType with nullable=true, sortOrder=List('a ASC NULLS FIRST) *** FAILED *** (72 milliseconds)
2025-11-06T19:59:48.4100453Z [info]   Exception thrown while executing Spark plan to calculate expected answer:
2025-11-06T19:59:48.4101154Z [info]    !ReferenceSort ['a ASC NULLS FIRST], true
2025-11-06T19:59:48.4101660Z [info]   +- Scan ExistingRDD[a#316981]
2025-11-06T19:59:48.4102044Z [info]   
2025-11-06T19:59:48.4102324Z [info]    == Exception ==
2025-11-06T19:59:48.4104862Z [info]    org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1190.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1190.0 (TID 3180) (d295a5912a94 executor driver): org.apache.comet.CometNativeException: Fail to process Arrow array with reason: Invalid argument error: RowConverter column schema mismatch, expected Timestamp(Microsecond, Some("America/Los_Angeles")) got Timestamp(Microsecond, Some("UTC")).
2025-11-08T19:38:36.1187839Z [info] - literal argument of named parameter in `sql()` *** FAILED *** (200 milliseconds)
2025-11-08T19:38:36.1190376Z [info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 84.0 failed 1 times, most recent failure: Lost task 0.0 in stage 84.0 (TID 96) (60d7cdca4a54 executor driver): org.apache.comet.CometNativeException: Invalid argument error: Invalid comparison operation: Timestamp(Microsecond, Some("America/Los_Angeles")) < Timestamp(Microsecond, Some("UTC"))

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove avatar Nov 06 '25 20:11 andygrove

We see the same issue in https://github.com/apache/datafusion-comet/issues/2649

andygrove avatar Nov 06 '25 23:11 andygrove