s3proxy icon indicating copy to clipboard operation
s3proxy copied to clipboard

Feature/overlay middleware

Open SwitchTV-BenBettridge opened this issue 3 years ago • 2 comments

This adds a new Middleware that implements writes using a local BlobStore and passes reads through to an upstream BlobStore using some simple logic:

  • Read requests are returned from the local BlobStore if the key exists, otherwise from the upstream bucket.
  • Requests that list container contents are merged between the local and upstream BlobStore
  • Writes are written to the local BlobStore
  • Delete's are implemented by creating a file in the local BlobStore which indicates the key should be removed from results.

Notes:

  • I am not a Java Developer (at least, not in the most recent decade) and this code might not follow all, or many, best practices. Particularly any features used that weren't a part of Java 7.
  • This is not production-ready. It was designed for development and testing purposes only. A non-exhaustive list of things that would be needed to be Production Ready are:
    • Use a more robust mechanism to generate "Not Found" errors other than trying to load a hopefully non-existent file.
    • The local BlobStore might need a mechanism to limit the amount of space it can consume, and how to handle the case where that space is exhausted.

This addresses a common use-case where developers need to test or replicate issues against production storage without actually writing anything back.

SwitchTV-BenBettridge avatar Feb 25 '22 01:02 SwitchTV-BenBettridge

This seems like a good feature -- sorry I haven't had time to look at it. Could you squash the commits and resolve the conflicts before I start reviewing it?

gaul avatar Jan 19 '23 03:01 gaul

@gaul - Good to go

bbetter173 avatar Feb 02 '23 22:02 bbetter173