mountpoint-s3
mountpoint-s3 copied to clipboard
awslabs
Add support for minio and other non-AWS S3 object stores
Tell us more about this new feature.
It would be very useful if mountpoint-s3 could support other object store backends and not just AWS. For example MINIO is a very popular object store that is, in large part, compatible with the S3 API. Thanks!
Thanks for the feedback! We tried to capture our thinking here in the README:
Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3.
In other words, Mountpoint probably works today against other S3-like services — you can point the --endpoint-url config anywhere you want, just like with the AWS CLIs — but we're not able to officially test or support that. I'm also not sure about the status of newer S3 features like additional checksums in other services. It's possible those will break, or at least need extra configuration to work around, as we adopt them for Mountpoint in the future.
Oh thanks @jamesbornholt! I missed the section of the README that you reference above and got fixated on the earlier comments on AWS credential files. I'll try and find some time to do a smoke test of mountpoint against minio and if I get anywhere I will post an update.
I totally get that your primary focus is AWS S3!!
Oh thanks @jamesbornholt! I missed the section of the README that you reference above and got fixated on the earlier comments on AWS credential files. I'll try and find some time to do a smoke test of mountpoint against minio and if I get anywhere I will post an update.
I totally get that your primary focus is AWS S3!!
Maybe you can also try this project , It is used to mount object storage locally and supports minio
@Waterman1997 juicefs is not just a tool that mounts an existing S3 bucket. It's an entire filesystem that stores its data in S3 in its own format.
@tcassaert , maybe https://github.com/s3fs-fuse/s3fs-fuse is what you are looking for
I think this project has some key advantages that other projects do not have:
- Focus on performance
- Written in a memory safe language, improving security and stability
I totally get that this is mainly used for AWS S3, but maybe we as the community could introduce some kind of tests towards other now AWS object stores (like Minio or RGW), this would be great. I am not a rust developer myself and I know it's a difficult task, but it would make this project infinitely more appealing.
We tested this with netapp storageGRID and this works. I don't think any particular testing is necessarily needed. Rather, features should be added - keeping in mind this use case. E.g adding a dependency on cloudwatch or IAM without an option to use without - will break onprem use cases.
https://community.netapp.com/t5/Tech-ONTAP-Blogs/Mountpoint-for-Amazon-S3-alpha-release-with-StorageGRID/ba-p/442993
Mapping these requirements to Mountpoint’s core mission and tenets led us to not prioritize this feature this year.
