datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Extended explain info missing fallback reason

Open andygrove opened this issue 11 months ago • 0 comments

Describe the bug

I added a new test in https://github.com/apache/datafusion-comet/pull/1308 and it fails on Spark 4.0 with:

2025-01-20T02:01:13.8889203Z - array_remove - fallback for unsupported type struct *** FAILED *** (172 milliseconds)
2025-01-20T02:01:13.8891189Z   Extended explain diff did not contain [data type not supported: ArrayType(StructType(StructField(_1,BooleanType,true),StructField(_2,ByteType,true)),false)]. Diff: metColumnarToRow
2025-01-20T02:01:13.8893645Z      +- CometScan parquet [_1#562725,_2#562726] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/__w/datafusion-comet/datafusion-comet/spark/target/tmp/spark-f50..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_1:boolean,_2:tinyint>

The fallback reason is not getting reported in the extended explain info. For now I added a parameters so that I can disable the extended info check.

Steps to reproduce

Modify the test added in https://github.com/apache/datafusion-comet/pull/1308 to enable the extended info check.

Expected behavior

No response

Additional context

No response

andygrove avatar Jan 20 '25 15:01 andygrove