S3 as a BackEnd

[blochsbek]

I'm struggling to deploy Inception as a high availability instance for end users, due to the nature of requiring a drive backed volume.

Proposed Solution:

• Utilize AWS S3 as a storage medium for documents/projects/other required information that cannot be stored in the database
• Store all other items in the database

Alternatives would be to store the data inside the database as encoded blob objects, but it would be tough.

This is designed to support inception running in Kubernetes and for high availability, more of a corporate setting. Other cloud storage providers could also be useful, (Azure, GCP etc)

[reckart]

This is not a storage issue, is it? This is an issue about you wanting to attach multiple instances of INCEpTION to the same storage/db, right?

Note that INCEpTION internally has a bunch of caches and exclusivity assumptions that could be violated if multiple instances would access the same data - so just solving the data storage would probably not be sufficient.

Also, storing the data on a non-local medium like S3 would likely slow down access a lot, negatively impacting the response times for users.

[blochsbek]

Right, exactly, I'd love to be able to run inception even if one pod goes down. I think it's more of a can we store pointers to files on S3 and download as necessary for annotation. Most often, it's not going to be required to have all the documents on disk.