Andrew Gaul
Andrew Gaul
Presently S3Proxy allows changing the storage class to and from Glacier. For the latter, a middleware could mimic the 3-5 hour latency, configurable via properties.
Protect against accidental overwrites or delete with an immutable blob middleware. This implementation would do a HEAD before every PUT or DELETE request, denying the former if an object already...
Allow S3 applications to use HDFS. jclouds has some long-bitrotted example of this: https://github.com/jclouds/jclouds-examples/tree/master/blobstore-hdfs There are a couple ways to do this, including using the Java bindings: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html or the...
S3Proxy can use jclouds `ForwardingBlobStore` to mirror write operations to two storage backends. Failure to write to either store should fail the client request. Note that this will leave the...
Some client workloads consist of reading many small objects, often sequentially in object listing order. S3Proxy could prefetch these to save time when using remote blobstores.
Breaking the almost 3,000 line `S3ProxyHandler` into component authentication, bucket, object, and multipart classes will make it easier to understand.
[S3Guard](https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/) adds strong consistency to Amazon S3 by storing all metadata in a DynamoDB database, similar to [S3mper](https://medium.com/netflix-techblog/s3mper-consistency-in-the-cloud-b6a1076aa4f8). S3Proxy could do something similar via the filesystem or JDBC blobstore.
S3Proxy middlewares presently require writing Java code and integrating them into the main S3Proxy binary during compile time. Instead S3Proxy should dynamically load scripts at run time to allow casual...
This would enable data privacy on backends without encryption, e.g., Rackspace Cloud Files, as well as improve privacy on backends with it, e.g., Amazon S3, due to handling the private...