delta-io
chore: bump arrow and datafusion
Description
~~Using the latest arrow (18) and datafusion (10) dependencies.~~
supersedes #666
This PR moves most file path handling to use the Path abstraction from object_store. We do so by way of moving our internal StorageBackend behind a new DeltaObjectStore, which implements ObjectStore. The delta store handles all path be always relative to the table root, consistent with how they are treated in the log. When listing URIs, always full path including the scheme is returned - this introduces a bit of nuisance in some of the code, but I thought to prioritize consistency.
- For local file system the table root folder, has to exist now, but can be empty.
- also bumps arrow (18) and datafusion (10)
Related Issue(s)
Documentation
@houqp @wjones127 - latest datafusion is already using the Path abstraction from object_strore_rs as well as the specific "view" on what an object store is. For the unix systems quickly hacked something together to make it work. For windows paths its still failing.
However since #696 has also been raised, I was wondering if we should start to integrate with the object_Store_rs crate. While its too early to use the object store itself just yet, maybe we should start to introduce the Path abstraction along with the implications on how we handle paths in general.
Here I could either switch the focus of this PR - or maybe skip datafusion tests on windows and do a follow up.
Of course I could also go into handling windows paths, but right now this feels very temporary ... Thoughts?
While its too early to use the object store itself just yet, maybe we should start to introduce the Path abstraction along with the implications on how we handle paths in general.
If it seems feasible, that seems like it's likely a good idea. I've been thinking trying to integrate object_store_rs entirely in 1 PR would be too much. So if we can migrate the Paths part first, I think that's a good idea.
