datafusion-comet
datafusion-comet copied to clipboard
More types in `try_*`
What is the problem the feature request solves?
Some data types seem to be missing in the current implementation of try_*. I might be a bit late to the party 😅 but I didn’t find an open issue tracking this. Comes from here https://github.com/apache/datafusion-comet/pull/2073
Describe the potential solution
Support for more data types, similar to how it was done in Sail. Here are some example tests from Spark showing expected behavior:
>>> df = spark.sql("""
... SELECT
... try_add(DATE '2015-09-30', 1) as d1,
... try_add(DATE '2000-01-01', 366) as d2,
... try_add(DATE '2021-01-01', 1) as d3,
... try_add(NULL, 100) as d4
... """)
>>> df.show()
+----------+----------+----------+----+
| d1| d2| d3| d4|
+----------+----------+----------+----+
|2015-10-01|2001-01-01|2021-01-02|NULL|
+----------+----------+----------+----+
>>> df = spark.sql("""
... SELECT
... try_add(DATE '2015-01-31', INTERVAL 1 MONTH) as d1,
... try_add(DATE '2020-02-29', INTERVAL 12 MONTH) as d2,
... try_add(NULL, INTERVAL 3 MONTH) as d3
... """)
>>> df.show()
+----------+----------+----+
| d1| d2| d3|
+----------+----------+----+
|2015-02-28|2021-02-28|NULL|
+----------+----------+----+
>>> df = spark.sql("""
... SELECT
... try_add(DATE '2000-07-31', INTERVAL -1 MONTH) as d1,
... try_add(DATE '2021-01-31', INTERVAL -1 MONTH) as d2
... """)
>>> df.show()
+----------+----------+
| d1| d2|
+----------+----------+
|2000-06-30|2020-12-31|
+----------+----------+
>>> df = spark.sql("""
... SELECT
... try_add(INTERVAL '1' YEAR, INTERVAL '2' YEAR) as result
... """)
>>> spark.conf.set("spark.sql.session.timeZone", "UTC")
>>> df = spark.sql("""
... SELECT
... try_add(TIMESTAMP '2021-01-01 00:00:00', INTERVAL 1 DAY) as result
... """)
>>> df.show(truncate=False)
+-------------------+
|result |
+-------------------+
|2021-01-02 00:00:00|
+-------------------+
Additional context
Implementation in Sail for reference: https://github.com/lakehq/sail/pull/635
Thank you very much for filling this issue. I will be working on this once my ANSI changes are completed