datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

External error: Arrow error: Invalid argument error: must either specify a row count or at least one column

Open andygrove opened this issue 1 month ago • 0 comments

Describe the bug

Spark SQL test failure when I enable spark.comet.sparkToColumnar.enabled.

[info] - SPARK-7595: Window will cause resolve failed with self join *** FAILED *** (116 milliseconds)
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 158.0 failed 1 times, most recent failure: Lost task 0.0 in stage 158.0 (TID 108) (10.0.0.118 executor driver): org.apache.comet.CometNativeException: External error: Arrow error: Invalid argument error: must either specify a row count or at least one column
[info] 	at org.apache.comet.Native.executePlan(Native Method)
[info] 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:151)
[info] 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:150)
[info] 	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:212)
[info] 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:150)
[info] 	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
[info] 	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:148)
[info] 	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:204)
[info] 	at org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriter.write(CometNativeShuffleWriter.scala:110)
[info] 	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove avatar Nov 06 '25 19:11 andygrove