datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

feat: Support ANSI mode for round

Open andygrove opened this issue 1 year ago • 3 comments

What is the problem the feature request solves?

Comet does not support ANSI mode for round.

Create test data

val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")

Test with ANSI enabled

Comet does not throw an exception but Spark does.

scala> spark.conf.set("spark.comet.ansi.enabled", true)

scala> spark.conf.set("spark.sql.ansi.enabled", true)

scala> spark.conf.set("spark.comet.enabled", true)

scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+------------+
|          a|round(a, -1)|
+-----------+------------+
| 2147483647| -2147483646|
|-2147483648|  2147483646|
+-----------+------------+


scala> spark.conf.set("spark.comet.enabled", false)

scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:37 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:38 ERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 11) ==
select a, round(a,-1) from t
          ^^^^^^^^^^^

Describe the potential solution

No response

Additional context

No response

andygrove avatar May 23 '24 20:05 andygrove

I will work on this one

vidyasankarv avatar May 24 '24 06:05 vidyasankarv

Had to take a temporary break, Will resume this if no one else is already looking into this. Thank you

vidyasankarv avatar Jun 19 '24 12:06 vidyasankarv

take

raulcd avatar Oct 01 '24 17:10 raulcd

Working on this

coderfender avatar Oct 10 '25 01:10 coderfender