datafusion-comet
datafusion-comet copied to clipboard
Review use of hard-coded UTC assumptions
What is the problem the feature request solves?
This code pattern worries me. We should fall back to Spark if we do not know the timezone.
spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala: .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala: .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala: castBuilder.setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/structs.scala: .setTimezone(expr.timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
Describe the potential solution
No response
Additional context
No response
Also serde.rs to_arrow_datatype has hard-coded UTC.