seafowl icon indicating copy to clipboard operation
seafowl copied to clipboard

Published 20 hours ago • verified publisher icon splitgraph

Local object cache for S3

Open mildbyte opened this issue 2 years ago • 0 comments

When using an S3-compatible object store, DataFusion / Seafowl currently download the object every time they need to scan through it. We should let users dedicate some scratch space on a Seafowl instance as object cache. Potential options:

  • write a wrapper around the ObjectStore trait that proxies an upstream object store and uses a local FS as cache
  • use https://github.com/kahing/goofys / + catfs OR s3fs?

mildbyte avatar Aug 03 '22 13:08 mildbyte