datafusion-comet
datafusion-comet copied to clipboard
Spark SQL test failures due to timestamp mismatch when LocalTableScan is native
Describe the bug
I saw this in Spark SQL tests with spark.comet.sparkToColumnar.enabled=true.
2025-11-06T19:59:48.4097661Z [info] - sorting on TimestampType with nullable=true, sortOrder=List('a ASC NULLS FIRST) *** FAILED *** (72 milliseconds)
2025-11-06T19:59:48.4100453Z [info] Exception thrown while executing Spark plan to calculate expected answer:
2025-11-06T19:59:48.4101154Z [info] !ReferenceSort ['a ASC NULLS FIRST], true
2025-11-06T19:59:48.4101660Z [info] +- Scan ExistingRDD[a#316981]
2025-11-06T19:59:48.4102044Z [info]
2025-11-06T19:59:48.4102324Z [info] == Exception ==
2025-11-06T19:59:48.4104862Z [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1190.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1190.0 (TID 3180) (d295a5912a94 executor driver): org.apache.comet.CometNativeException: Fail to process Arrow array with reason: Invalid argument error: RowConverter column schema mismatch, expected Timestamp(Microsecond, Some("America/Los_Angeles")) got Timestamp(Microsecond, Some("UTC")).
2025-11-08T19:38:36.1187839Z [info] - literal argument of named parameter in `sql()` *** FAILED *** (200 milliseconds)
2025-11-08T19:38:36.1190376Z [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 84.0 failed 1 times, most recent failure: Lost task 0.0 in stage 84.0 (TID 96) (60d7cdca4a54 executor driver): org.apache.comet.CometNativeException: Invalid argument error: Invalid comparison operation: Timestamp(Microsecond, Some("America/Los_Angeles")) < Timestamp(Microsecond, Some("UTC"))
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
We see the same issue in https://github.com/apache/datafusion-comet/issues/2649