spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

[BUG] `array_min` does not fall back to CPU when `hasNan = true`

Open HaoYang670 opened this issue 3 years ago • 1 comments

Describe the bug A clear and concise description of what the bug is.

Steps/Code to reproduce bug

scala> spark.conf.set("spark.rapids.sql.hasNans","true")

scala> val df = Seq((Seq(1.0, 2.0, 3.0)), (Seq(1.1, 2.1, Double.NaN))).toDF("a")
df: org.apache.spark.sql.DataFrame = [a: array<double>]

scala> df.select(array_min($"a")).collect
22/08/16 16:15:32 WARN GpuOverrides: 
! <LocalTableScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.LocalTableScanExec
  @Expression <AttributeReference> array_min(a)#6 could run on GPU

res3: Array[org.apache.spark.sql.Row] = Array([1.0], [1.1])

array_min should runs on CPU but not GPU

Expected behavior A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

  • Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
  • Spark configuration settings related to the issue

Additional context Related to #6330 and #6317

HaoYang670 avatar Aug 16 '22 08:08 HaoYang670

After removing the hasNan config fromGpuArrayMin(#6317), we can close this issue without fixing it.

HaoYang670 avatar Aug 17 '22 05:08 HaoYang670