datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

feat: Support `array<binary>` for `array_reverse`

Open comphead opened this issue 1 month ago • 0 comments

What is the problem the feature request solves?

Followup on #2612

We observe incorrect behavior for array_reverse if datatype is array<binary>. Currently we fallback to Spark and we need to support array<binary> for native execution

Repro issue

test("native reader - array_reverse") {
    withSQLConf(
      CometConf.COMET_EXEC_ENABLED.key -> "true",
      SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
      CometConf.COMET_ENABLED.key -> "true",
      CometConf.COMET_EXPLAIN_FALLBACK_ENABLED.key -> "true",
      CometConf.COMET_NATIVE_SCAN_IMPL.key -> "native_iceberg_compat") {

      testSingleLineQuery(
        """
          |SELECT array(
          |  cast(null as array<binary>),
          |  cast(array() as array<binary>)
          |) AS a
          |""".stripMargin,
        "select reverse(a) from tbl", debugCometDF = df => df.printSchema())
    }
  }

!== Correct Answer - 1 ==                  == Spark Answer - 1 ==
 struct<reverse(a):array<array<binary>>>   struct<reverse(a):array<array<binary>>>
![ArrayBuffer(WrappedArray(), null)]       [ArrayBuffer(WrappedArray(), WrappedArray())]

https://github.com/apache/datafusion-comet/issues/2612#issuecomment-3508492517

Describe the potential solution

No response

Additional context

No response

comphead avatar Nov 11 '25 19:11 comphead