datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Review use of hard-coded UTC assumptions

Open andygrove opened this issue 1 month ago • 1 comments

What is the problem the feature request solves?

This code pattern worries me. We should fall back to Spark if we do not know the timezone.

spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala:                .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala:                .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:        castBuilder.setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/structs.scala:              .setTimezone(expr.timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")

Describe the potential solution

No response

Additional context

No response

andygrove avatar Nov 07 '25 18:11 andygrove

Also serde.rs to_arrow_datatype has hard-coded UTC.

andygrove avatar Nov 07 '25 22:11 andygrove