iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

test: upgrade DataFusion 51 and Arrow 57

Open mbutrovich opened this issue 4 weeks ago • 2 comments

DataFusion is preparing for their 51.0.0 release, so this is a test PR to see how CI goes with main on a recent commit that likely reflects what DF 51 will be. Marking as draft since we wouldn't want this merged until crates are available and the community discusses the changes.

Which issue does this PR close?

  • N/A.

What changes are included in this PR?

  • Bump DF to 51
  • Bump Arrow/Parquet to 57
  • Page index tests are skipped right now because of API changes in Arrow-rs. I will have to revisit what we can do to fix these since they were relying on a pretty low-level API
  • Toolchain bump to support DF's 1.88. I need to understand if this would be within iceberg-rust's MSRV policy

Are these changes tested?

Creating this to run CI.

mbutrovich avatar Nov 07 '25 21:11 mbutrovich

i can probably help with the page index changes

alamb avatar Nov 08 '25 09:11 alamb

It was green at one point, so I think from DataFusion's standpoint DF 51 is okay. This branch quickly fell behind with merge conflicts. I'll update it once DF 51 crates are available and we want to consider upgrading for real.

mbutrovich avatar Nov 13 '25 13:11 mbutrovich