web-monitoring-processing icon indicating copy to clipboard operation
web-monitoring-processing copied to clipboard

Look into obstore instead of cloudpathlib

Open Mr0grog opened this issue 1 month ago • 0 comments
trafficstars

I really like how cloudpathlib gives gives us a fairly transparent way to interchangeably handle local and S3 paths for writing files. BUT there is a fancy new kid on the block for dealing with S3 style object stores that we should take a look at: obstore (nice blog post: https://developmentseed.org/blog/2025-08-01-obstore/).

At a glance, I know the Development Seed folks know their stuff and deal with a lot of complex object store scenarios, so this is likely to be exactly as advertised: robust, light, speedy, and reasonably well thought out. That said, it sounds like a Pathlib-style API is an explicit non-goal. I suspect we may be better off sticking with cloudpathlib since we are mostly not dealing with large files, partial/range reads, etc. but do get huge benefit from the Pathlib interface. Worth spending more than a few minutes on a deeper dive, though.

Mr0grog avatar Sep 30 '25 23:09 Mr0grog