datafusion-comet
datafusion-comet copied to clipboard
Extended explain info missing fallback reason
Describe the bug
I added a new test in https://github.com/apache/datafusion-comet/pull/1308 and it fails on Spark 4.0 with:
2025-01-20T02:01:13.8889203Z - array_remove - fallback for unsupported type struct *** FAILED *** (172 milliseconds)
2025-01-20T02:01:13.8891189Z Extended explain diff did not contain [data type not supported: ArrayType(StructType(StructField(_1,BooleanType,true),StructField(_2,ByteType,true)),false)]. Diff: metColumnarToRow
2025-01-20T02:01:13.8893645Z +- CometScan parquet [_1#562725,_2#562726] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/__w/datafusion-comet/datafusion-comet/spark/target/tmp/spark-f50..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_1:boolean,_2:tinyint>
The fallback reason is not getting reported in the extended explain info. For now I added a parameters so that I can disable the extended info check.
Steps to reproduce
Modify the test added in https://github.com/apache/datafusion-comet/pull/1308 to enable the extended info check.
Expected behavior
No response
Additional context
No response