MrCroxx
discuss: Add foyer-opendal as new device or storage
Hi, thank you very much for the excellent foyer library! I'm starting this discussion to explore the idea of adding foyer-opendal as a new device or storage option.
Possible services that could be used:
- monoiofs
- redis
- other in-memory caches like
moka
Hi @Xuanwo , thanks for following!
TBH, I've been considering integrating with opendal for a long time. I've tried once but was interrupted by other high-priority work.
About the idea, IIUC, foyer is supposed to be a cache layer for opendal, instead of using opendal as a device (storage layer) in foyer, because foyer is used to reduce remote storage access overhead and optimize io with special features of fs/block over disk. Do you agree with it or have I missed something?
Thanks a lot for the quick feedback.
About the idea, IIUC, foyer is supposed to be a cache layer for opendal, instead of using opendal as a device (storage layer) in foyer, because foyer is used to reduce remote storage access overhead and optimize io with special features of fs/block over disk. Do you agree with it or have I missed something?
Your understanding is correct but incomplete.
This can work both ways: Foyer can serve as OpenDAL's cache layer, optimizing the performance of reading data from slow storage like S3. At the same time, OpenDAL can serve as Foyer's device (or other appropriate abstraction), reducing overhead by storing data in other existing storage services like Redis, Moka, or Memcached.
I use opendal in my python project, having foyer's cache and hybridcache as options for object store caching would be very nice :+1:
