zarr-developers
Should all stores have a cache? Should arrays and groups have a cache?
The current implementation of caching for stores is via a store-like wrapper that encapsulates an existing store. An alternative approach would be to add caching to the store API, which would basically give every existing store a cache.
In keeping with the idea that zarr is designed for large, remote datasets, I think we should consider caching a basic and necessary feature of storage backends. This argues for making caching a feature of the base storage layer, i.e. adding caching to BaseStore or Store. We could start by taking all the cache-specific methods defined for LRUStoreCache and moving that functionality to a base caching class / caching protocol, then add a cache keyword argument + class attribute to Store which must be an instance of the Cache class or an object that implements the cache protocol. We should also ensure that there is a DummyCache defined that does nothing so that users who want no caching can achieve that.
Beyond that, I don't have any clear ideas what what would change for specific subclasses of Store. Perhaps we would have to touch every single subclass, in which case I think the effort would almost certainly be worth it for the performance gains.
A secondary question is whether Array and Group should also have a built-in cache that follows the same API outlined above. This would remove the need for defining caching logic in Attrs. Also, for slow compression routines, it may be advantageous to cache decompressed chunks in Array instances.
Curious to hear what people think about this.
I like this! Came here to open a feature request for disk cache support; glad I found this first.
