Should we try to use Obstore for reading files?

[norlandrhagen]

Do folks think it's worth replacing the fsspec file handler_FsspecFSFromFilepath with an obstore based one? I think we could get a lot of benefits from how obstore handles credentials.

cc @TomNicholas @maxrjones @abarciauskas-bgse @sharkinsspatial

As an experiment I took @sharkinsspatial's HDF reader, and replaced the _FsspecFSFromFilepath with an obstore reader and the test suite passed. I'm sure there would be a lot of other places to update.

obsreader = ObstoreReader(filepath = path)
f = h5py.File(obsreader.reader(), mode="r")

[maxrjones]

I would love this, as much as anything because it's a great opportunity to stress-test obstore a bit 😄

I would be most excited to open an obstore-backed alternative to https://github.com/zarr-developers/VirtualiZarr/pull/458 because I think the abstractions in obstore would work really well for ManifestStore. It'd just be helpful for https://github.com/zarr-developers/zarr-python/pull/1661 to get merged first. I'm going to ping the maintainers to ask for a review after obstore v0.5.0 comes out, which adds some helpful conditional write stuff.

[TomNicholas]

I would be most excited to open an obstore-backed alternative to https://github.com/zarr-developers/VirtualiZarr/pull/458

That did not occur to me but it's an intriguing idea!

[maxrjones]

#601 was the final step for this issue