Andy Grove

Hi @dharanad. All of the issues here are important for Spark 4.0 support. Feel free to pick up any issues that are not already being worked on

This recent blog post is worth a read. It talks about some performance issues with musl's default allocator and how switching to jemalloc helps. https://blog.sdf.com/p/fast-development-in-rust-part-one

I do have xcode installed. I tried with openjdk 11 as well. ``` % java --version openjdk 11.0.22 2024-01-16 OpenJDK Runtime Environment Homebrew (build 11.0.22+0) OpenJDK 64-Bit Server VM Homebrew...

I tried installing protobuf with homebrew as well as manually. In either case, maven runs the downloaded one, which is not really an arm64 build even though it pretends to...

This is no longer an issue for me and we have not had other reports of this happening, so will close this

Comet is now using 39.0.0-rc1 (and will upgrade to 39.0.0 when it is available, probably early next week) so I think we can close this issue now

Now that Comet supports DataFusion's `DataSourceExec` (when `native_datafusion` scan is enabled) it should be much easier to support `delta-rs`.

We now have three Parquet scan implementations - `native_comet` and `native_iceberg_compat` which are hybrid JVM and Rust, and we also have `native_datafusion` which is pure Rust (uses DataFusion's `DataSourceExec`). `native_iceberg_compat`...

The short term roadmap is now available in this blog post: https://datafusion.apache.org/blog/2024/07/20/datafusion-comet-0.1.0/

This issues appears to be resolved, so I will close this issue. Thanks @okue for reporting it. ``` scala> Seq(15901).toDF("dt").write.parquet("/tmp/dt") 24/09/19 10:45:43 INFO core/src/lib.rs: Comet native library version 0.3.0 initialized...