datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Spark SQL test "write null values" fails with Arrow error "Accounted size went negative"

Open andygrove opened this issue 11 months ago • 0 comments

Describe the bug

When we fix https://github.com/apache/datafusion-comet/issues/1248 we find a new bug that causes the following Spark SQL test to fail:

- write null values *** FAILED *** (122 milliseconds)
2025-01-01T22:28:02.8649715Z Caused by: java.lang.IllegalArgumentException: Accounted size went negative.
2025-01-01T22:28:02.8650963Z 	at org.apache.comet.shaded.arrow.util.Preconditions.checkArgument(Preconditions.java:141)
2025-01-01T22:28:02.8652364Z 	at org.apache.comet.shaded.arrow.memory.Accountant.releaseBytes(Accountant.java:221)
2025-01-01T22:28:02.8653779Z 	at org.apache.comet.shaded.arrow.memory.RootAllocator.releaseBytes(RootAllocator.java:29)
2025-01-01T22:28:02.8655201Z 	at org.apache.comet.shaded.arrow.memory.AllocationManager.release(AllocationManager.java:154)
2025-01-01T22:28:02.8656578Z 	at org.apache.comet.shaded.arrow.memory.BufferLedger.decrement(BufferLedger.java:158)
2025-01-01T22:28:02.8658033Z 	at org.apache.comet.shaded.arrow.memory.BufferLedger.release(BufferLedger.java:125)
2025-01-01T22:28:02.8659503Z 	at org.apache.comet.shaded.arrow.memory.BufferLedger.release(BufferLedger.java:105)
2025-01-01T22:28:02.8660742Z 	at org.apache.comet.shaded.arrow.memory.ArrowBuf.close(ArrowBuf.java:1045)
2025-01-01T22:28:02.8661993Z 	at org.apache.comet.shaded.arrow.c.ArrowSchema.close(ArrowSchema.java:134)
2025-01-01T22:28:02.8663252Z 	at org.apache.comet.shaded.arrow.c.ArrowImporter.importField(ArrowImporter.java:47)
2025-01-01T22:28:02.8664510Z 	at org.apache.comet.shaded.arrow.c.ArrowImporter.importVector(ArrowImporter.java:53)
2025-01-01T22:28:02.8665809Z 	at org.apache.comet.vector.NativeUtil.$anonfun$importVector$1(NativeUtil.scala:191)
2025-01-01T22:28:02.8667257Z 	at org.apache.comet.vector.NativeUtil.$anonfun$importVector$1$adapted(NativeUtil.scala:184)
2025-01-01T22:28:02.8668467Z 	at scala.collection.immutable.Range.foreach(Range.scala:158)
2025-01-01T22:28:02.8669535Z 	at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:184)
2025-01-01T22:28:02.8670792Z 	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:164)
2025-01-01T22:28:02.8672117Z 	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:92)
2025-01-01T22:28:02.8673399Z 	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:105)

Steps to reproduce

See https://github.com/apache/datafusion-comet/pull/1209

Expected behavior

No response

Additional context

No response

andygrove avatar Jan 09 '25 12:01 andygrove