Andy Grove

@RaghavendraGanesh We now have a 0.7.0 image for Spark 3.5.4. Perhaps that helps? https://hub.docker.com/r/apache/datafusion-comet/tags

> Hello [@andygrove](https://github.com/andygrove) , this sounds like an interesting problem I'd be keen to work on. I think I might need some guidance to navigate it effectively. Would you be...

I updated the issue with Claude's suggestions for implementing this feature.

@jinwenjie123 I would recommend starting off by using a pre-built JAR which contains native binaries for multiple architectures. https://datafusion.apache.org/comet/user-guide/installation.html#using-a-published-jar-file

We use Docker images when building these jars. Our release process is documented at https://github.com/apache/datafusion-comet/tree/main/dev/release#readme

I created https://github.com/apache/datafusion/pull/15947 to add Comet's `hex` function to `datafusion-spark`. This involved converting it from a `PhysicalExpr` to a `ScalarUDFImpl`. Also, I created https://github.com/apache/datafusion-comet/pull/1711 to use the `expm1` function from...

https://github.com/apache/datafusion-comet/pull/1711 is now merged, so Comet is now using the datafusion-spark crate

I will keep this issue open until we actually remove an expression from comet and use a version from data fusion-spark

@parthchandra @mbutrovich fyi

I'm looking into the core3 `row index generation` errors. At least one of them is failing with NPE in Comet code: ``` Caused by: java.lang.NullPointerException at org.apache.comet.parquet.NativeBatchReader.nextBatch(NativeBatchReader.java:427) at org.apache.comet.parquet.NativeBatchReader.nextKeyValue(NativeBatchReader.java:373) ```...