s3proxy HDFS storage backend

Allow S3 applications to use HDFS. jclouds has some long-bitrotted example of this:

https://github.com/jclouds/jclouds-examples/tree/master/blobstore-hdfs

There are a couple ways to do this, including using the Java bindings:

https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html

or the REST API:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html https://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/

Mar 01 '18 05:03 gaul

@gaul I see the jclouds hdfs example link is no longer working at HEAD but is visible at 1.8.1. This would be very useful backend for an S3 server to support. Any chance you are going to pick this up again anytime soon ?

May 16 '23 21:05 gkalele

I think it would be better to abandon the jclouds HDFS code and to write a small BlobStore implementation that calls the Hadoop Java bindings. I don't have the time to work on this myself but I think you could hack up a minimal implementation in a few days. Integrating this into S3Proxy is probably best done without proper jclouds support which raises questions about how the provider registration would work but this is something that should be solvable.

May 17 '23 05:05 gaul