rustus
rustus copied to clipboard
s3rius
Implement S3 data storage.
Create S3 Data storage to be able to store files on S3.
The main idea is to create many file chunks 5MB each and concatenate them after upload is finished.
It's quite a challenge. I won't do it for now. For all people who want to work with S3 as a primary storage, you can use s3fs-fuse to mount s3 in a directory.
Would it be more straightforward if the implementation used the volume storage as scratch space but then pushed to s3 as part of finishing the upload?
Hi @Kaelten, it took a long time, but this feature is finally coming in the next release! Currently we will only have a hybrid-s3 which works exactly as you said. The fully operating S3 storage would be implemented a little bit later.
Related PR: #108.
Hybrid-s3 storage is released in 0.6.0. S3 storage without storing files locally will come later. Leaving this feature open until new storage is released.
For an S3 only implementation, would not be S3 Multipart upload the solution? https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html (So concat should not be needed, that makes S3 for you)
Hi, @genofire. Indeed, that's exactly it. But the tricky part here is that minimum size for a chunk is 5Mb. For some systems it just don't work. That's why I implemented hybrid storage first. Hybrid storage doesn't have such a restriction.
Hi, @genofire. Yes, I guess it may work. I need to try it.
I would have no time in 1-2 weeks. If you feel that you might take this over, please do. If you don't, I'll take it as soon as I have more time.
Thank you for pointing this out.