delta-rs icon indicating copy to clipboard operation
delta-rs copied to clipboard

Published 20 hours ago • verified publisher icon delta-io

chore: datafusion 34, arrow & parquet 49

Open emcake opened this issue 1 year ago • 5 comments

Description

Upgrades DataFusion and Arrow to v34 and v49 respectively. As a casualty we also need to upgrade object_store to v0.8.

Related Issue(s)

Arrow 49 contains a fix to allow for truncated statistics on binary columns (https://github.com/apache/arrow-rs/issues/5037) that I'd like to employ to fix #1805 .

emcake avatar Dec 19 '23 14:12 emcake

@emcake - thanks for taking care of this!

Seems there is one more clippy error to address :).

roeap avatar Dec 19 '23 16:12 roeap

I think this is now complete and up to date - there still appears to be some tests failing but I'm struggling to work out if these are flaky or real.

emcake avatar Dec 20 '23 23:12 emcake

@emcake the restore by datetime and a read test are super flaky..

ion-elgreco avatar Dec 20 '23 23:12 ion-elgreco

@emcake maybe object store 0.8 is causing the read failures, can you revert to object store 0.7?

ion-elgreco avatar Dec 23 '23 11:12 ion-elgreco

Object store is coupled to datafusion. Had a first glance, and it seems like fixable problem, but may require some work.

roeap avatar Dec 23 '23 11:12 roeap

Looks like @roeap beat me to it!

emcake avatar Jan 04 '24 11:01 emcake