iceberg-rust
iceberg-rust copied to clipboard
test: upgrade DataFusion 51 and Arrow 57
DataFusion is preparing for their 51.0.0 release, so this is a test PR to see how CI goes with main on a recent commit that likely reflects what DF 51 will be. Marking as draft since we wouldn't want this merged until crates are available and the community discusses the changes.
Which issue does this PR close?
- N/A.
What changes are included in this PR?
- Bump DF to 51
- Bump Arrow/Parquet to 57
- Page index tests are skipped right now because of API changes in Arrow-rs. I will have to revisit what we can do to fix these since they were relying on a pretty low-level API
- Toolchain bump to support DF's 1.88. I need to understand if this would be within iceberg-rust's MSRV policy
Are these changes tested?
Creating this to run CI.
i can probably help with the page index changes
It was green at one point, so I think from DataFusion's standpoint DF 51 is okay. This branch quickly fell behind with merge conflicts. I'll update it once DF 51 crates are available and we want to consider upgrading for real.