web-monitoring-processing
web-monitoring-processing copied to clipboard
Look into obstore instead of cloudpathlib
I really like how cloudpathlib gives gives us a fairly transparent way to interchangeably handle local and S3 paths for writing files. BUT there is a fancy new kid on the block for dealing with S3 style object stores that we should take a look at: obstore (nice blog post: https://developmentseed.org/blog/2025-08-01-obstore/).
At a glance, I know the Development Seed folks know their stuff and deal with a lot of complex object store scenarios, so this is likely to be exactly as advertised: robust, light, speedy, and reasonably well thought out. That said, it sounds like a Pathlib-style API is an explicit non-goal. I suspect we may be better off sticking with cloudpathlib since we are mostly not dealing with large files, partial/range reads, etc. but do get huge benefit from the Pathlib interface. Worth spending more than a few minutes on a deeper dive, though.