datafusion-comet
datafusion-comet copied to clipboard
feat: Support ANSI mode for round
What is the problem the feature request solves?
Comet does not support ANSI mode for round.
Create test data
val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
Test with ANSI enabled
Comet does not throw an exception but Spark does.
scala> spark.conf.set("spark.comet.ansi.enabled", true)
scala> spark.conf.set("spark.sql.ansi.enabled", true)
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+------------+
| a|round(a, -1)|
+-----------+------------+
| 2147483647| -2147483646|
|-2147483648| 2147483646|
+-----------+------------+
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:37 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:38 ERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 11) ==
select a, round(a,-1) from t
^^^^^^^^^^^
Describe the potential solution
No response
Additional context
No response
I will work on this one
Had to take a temporary break, Will resume this if no one else is already looking into this. Thank you
take
Working on this