datafusion-comet
datafusion-comet copied to clipboard
feat: Support `array<binary>` for `array_reverse`
What is the problem the feature request solves?
Followup on #2612
We observe incorrect behavior for array_reverse if datatype is array<binary>. Currently we fallback to Spark and we need to support array<binary> for native execution
Repro issue
test("native reader - array_reverse") {
withSQLConf(
CometConf.COMET_EXEC_ENABLED.key -> "true",
SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
CometConf.COMET_ENABLED.key -> "true",
CometConf.COMET_EXPLAIN_FALLBACK_ENABLED.key -> "true",
CometConf.COMET_NATIVE_SCAN_IMPL.key -> "native_iceberg_compat") {
testSingleLineQuery(
"""
|SELECT array(
| cast(null as array<binary>),
| cast(array() as array<binary>)
|) AS a
|""".stripMargin,
"select reverse(a) from tbl", debugCometDF = df => df.printSchema())
}
}
!== Correct Answer - 1 == == Spark Answer - 1 ==
struct<reverse(a):array<array<binary>>> struct<reverse(a):array<array<binary>>>
![ArrayBuffer(WrappedArray(), null)] [ArrayBuffer(WrappedArray(), WrappedArray())]
https://github.com/apache/datafusion-comet/issues/2612#issuecomment-3508492517
Describe the potential solution
No response
Additional context
No response