velox
velox copied to clipboard
Support timestamp and date types for Spark unix_timestamp function
Spark unix_timestamp and to_unix_timestamp support DateType, TimestampType, TimestampNTZType besides StringType, see https://github.com/apache/spark/blob/039fd13eacb1cef835045e3a60cebf958589e1a2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L1271.
This PR adds DateType and TimestampType support for them.
Deploy Preview for meta-velox canceled.
| Name | Link |
|---|---|
| Latest commit | 203baca3b44ee71646973a5a433f392cbcc3da7a |
| Latest deploy log | https://app.netlify.com/sites/meta-velox/deploys/67be92f8c341f80008cd75de |
Hey @rui-mo @PHILO-HE , could you please help review this?
This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!
@boneanxs Would you like to take a look at the fuzzer failure? Thanks.
Error Source: RUNTIME Error Code: INVALID_STATE Reason: Unable to convert timezone 'America/Los_Angeles' past 2037-11-01 09:00:00 Retriable: False Context: Top-level Expression: to_unix_timestamp(date_add(1974-03-13:DATE, c0))
Fixed it by adding to skipFunctions like from_unixtime
@boneanxs I assume this fuzzer failure will be fixed by https://github.com/facebookincubator/velox/pull/11975#discussion_r1967432119.
@kevinwilfong has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@kevinwilfong merged this pull request in facebookincubator/velox@e59fb46a184b99ae9dce690d910a8a6d57247b86.