Spark SQL test failures in native_datafusion scan

[andygrove]

Describe the bug

There are more than 100 failures in core2 suite that seem to have the same root cause:

- Spark vectorized reader - without partition data column - select only top-level fields *** FAILED *** (317 milliseconds)
[info]   0 did not equal 1 Found 0 file sources in dataframe, but expected WrappedArray(struct<address:string>) (SchemaPruningSuite.scala:874)

[mbutrovich]

core 1: Failed: Total 9134, Failed 55, Errors 0, Passed 9079, Ignored 292, Canceled 6
core 2: Failed: Total 2655, Failed 140, Errors 0, Passed 2515, Ignored 387
core 3: Failed: Total 1522, Failed 56, Errors 0, Passed 1466, Ignored 370, Canceled 16
hive 1: Failed: Total 2148, Failed 39, Errors 0, Passed 2109, Ignored 40, Canceled 4
hive 2: Passed: Total 19, Failed 0, Errors 0, Passed 19, Ignored 1, Canceled 4
hive 3: Failed: Total 1055, Failed 2, Errors 0, Passed 1053, Ignored 13, Canceled 4

Counts from https://github.com/apache/datafusion-comet/pull/1543 today. Will track here as we keep updating.

[mbutrovich]

core 1: Failed: Total 9138, Failed 46, Errors 0, Passed 9092, Ignored 292, Canceled 6
core 2: Failed: Total 2655, Failed 25, Errors 0, Passed 2630, Ignored 387
core 3: Failed: Total 1523, Failed 49, Errors 0, Passed 1474, Ignored 370, Canceled 16
hive 1: Failed: Total 2148, Failed 36, Errors 0, Passed 2112, Ignored 40, Canceled 4
hive 2: Passed: Total 19, Failed 0, Errors 0, Passed 19, Ignored 1, Canceled 4
hive 3: Failed: Total 1055, Failed 2, Errors 0, Passed 1053, Ignored 13, Canceled 4

Updated 3.5.5 numbers from 2 days ago. 158 failures total. https://github.com/apache/datafusion-comet/actions/runs/14478169989

[mbutrovich]

core 1: Failed: Total 9138, Failed 38, Errors 0, Passed 9100, Ignored 292, Canceled 6
core 2: Failed: Total 2655, Failed 23, Errors 0, Passed 2632, Ignored 387
core 3: Failed: Total 1523, Failed 49, Errors 0, Passed 1474, Ignored 370, Canceled 16
hive 1: Failed: Total 2148, Failed 34, Errors 0, Passed 2114, Ignored 40, Canceled 4
hive 2: Passed: Total 19, Failed 0, Errors 0, Passed 19, Ignored 1, Canceled 4
hive 3: Failed: Total 1055, Failed 2, Errors 0, Passed 1053, Ignored 13, Canceled 4

Updated 3.5.5 numbers from today. 146 failures total. https://github.com/apache/datafusion-comet/actions/runs/14643258190

[mbutrovich]

core 1: Failed: Total 9138, Failed 38, Errors 0, Passed 9100, Ignored 292, Canceled 6
core 2: Failed: Total 2655, Failed 23, Errors 0, Passed 2632, Ignored 387
core 3: Failed: Total 1473, Failed 3, Errors 0, Passed 1470, Ignored 370, Canceled 66
hive 1: Failed: Total 2148, Failed 18, Errors 0, Passed 2130, Ignored 40, Canceled 4
hive 2: Passed: Total 19, Failed 0, Errors 0, Passed 19, Ignored 1, Canceled 4
hive 3: Failed: Total 1055, Failed 2, Errors 0, Passed 1053, Ignored 13, Canceled 4

Updated 3.5.5 numbers from today. 84 failures total. https://github.com/apache/datafusion-comet/actions/runs/15001712282