spark-rapids
spark-rapids copied to clipboard
[BUG] `array_min` does not fall back to CPU when `hasNan = true`
Describe the bug A clear and concise description of what the bug is.
Steps/Code to reproduce bug
scala> spark.conf.set("spark.rapids.sql.hasNans","true")
scala> val df = Seq((Seq(1.0, 2.0, 3.0)), (Seq(1.1, 2.1, Double.NaN))).toDF("a")
df: org.apache.spark.sql.DataFrame = [a: array<double>]
scala> df.select(array_min($"a")).collect
22/08/16 16:15:32 WARN GpuOverrides:
! <LocalTableScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.LocalTableScanExec
@Expression <AttributeReference> array_min(a)#6 could run on GPU
res3: Array[org.apache.spark.sql.Row] = Array([1.0], [1.1])
array_min should runs on CPU but not GPU
Expected behavior A clear and concise description of what you expected to happen.
Environment details (please complete the following information)
- Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
- Spark configuration settings related to the issue
Additional context Related to #6330 and #6317
After removing the hasNan config fromGpuArrayMin(#6317), we can close this issue without fixing it.