velox icon indicating copy to clipboard operation
velox copied to clipboard

Support timestamp and date types for Spark unix_timestamp function

Open boneanxs opened this issue 1 year ago • 2 comments

Spark unix_timestamp and to_unix_timestamp support DateType, TimestampType, TimestampNTZType besides StringType, see https://github.com/apache/spark/blob/039fd13eacb1cef835045e3a60cebf958589e1a2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L1271.

This PR adds DateType and TimestampType support for them.

boneanxs avatar Sep 29 '24 09:09 boneanxs

Deploy Preview for meta-velox canceled.

Name Link
Latest commit 203baca3b44ee71646973a5a433f392cbcc3da7a
Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/67be92f8c341f80008cd75de

netlify[bot] avatar Sep 29 '24 09:09 netlify[bot]

Hey @rui-mo @PHILO-HE , could you please help review this?

boneanxs avatar Oct 08 '24 08:10 boneanxs

This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!

stale[bot] avatar Jan 28 '25 18:01 stale[bot]

@boneanxs Would you like to take a look at the fuzzer failure? Thanks.

Error Source: RUNTIME Error Code: INVALID_STATE Reason: Unable to convert timezone 'America/Los_Angeles' past 2037-11-01 09:00:00 Retriable: False Context: Top-level Expression: to_unix_timestamp(date_add(1974-03-13:DATE, c0))

Fixed it by adding to skipFunctions like from_unixtime

boneanxs avatar Feb 25 '25 07:02 boneanxs

@boneanxs I assume this fuzzer failure will be fixed by https://github.com/facebookincubator/velox/pull/11975#discussion_r1967432119.

rui-mo avatar Feb 25 '25 08:02 rui-mo

@kevinwilfong has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot avatar Feb 26 '25 18:02 facebook-github-bot

@kevinwilfong merged this pull request in facebookincubator/velox@e59fb46a184b99ae9dce690d910a8a6d57247b86.

facebook-github-bot avatar Feb 27 '25 17:02 facebook-github-bot