storage
storage copied to clipboard
supabase
Allow users to BYO S3-compatible storage backend
We should make it easy for users to export their objects. Also useful for backups etc. We can probably use the S3 replication functionality to do this.
Relevant discussion https://github.com/supabase/supabase/discussions/1930
Is there any update on this? The lack of backup-from-supabase-storage function is stopping me to use supabase storage.
Hi @d-mok, we couldn't find an easy way of doing this with the S3 replication functionality above since there there is a limit of 1,000 replication policies per bucket which doesn't work at our scale.
If you have any ideas on how to go about implementing this in a multi-tenant way, let me know!
@inian Hi, are there any other plans for syncing files out of Supabase Storage?
@inian Then, may be the direction to go is to allow users to provide their own S3 bucket? Like, a bring-your-own-bucket approach?
This would be perfect.
On Sun, Aug 8, 2021, 6:32 PM d-mok @.***> wrote:
@inian https://github.com/inian Then, may be the direction to go is to allow users to provide their own S3 bucket? Like, a bring-your-own-bucket approach?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/supabase/storage-api/issues/27#issuecomment-894900593, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGWU44AV4UUCIW7UDHU2HTT34V3VANCNFSM46R3X7HA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
What I'm thinking is to have a new replication section in the UI where you can enter an access key ID and secret access key for an AWS IAM user, and a bucket, and then we'll regularly replicate to that bucket. How does that sound?
What I'm thinking is to have a new replication section in the UI where you can enter an access key ID and secret access key for an AWS IAM user, and a bucket, and then we'll regularly replicate to that bucket. How does that sound?
So we (users) provide a S3 bucket, and supabase regularly backup everything to that bucket? That sounds good enough. But I think a better option is to allow user to bring-their-own-bucket. In this way, I can interact with my bucket both through supabase or via some other s3 client. It would be best if, for example, when I add a file to my bucket through another client, supabase can also register the change. This is consistent with the principle how supabase is dealing with Postgresql: we can use the postgresql database any way we want, and supabase will work accordingly.
I'd mirror what @d-mok is expressing. Bring your own S3 bucket is very useful for cases that data governance is a consideration as well. Usually for larger organizations have these challenges. It seems lately many SaaS offerings do provide a bring-your-own-bucket model.
Another vote in favor of bring-your-own-bucket: I think it would allow us to manage data transfer costs better (e.g. keeping transfers within AWS infrastructure in the same region and AZ) which is important for data/processing-heavy applications.
I opened a separate discussion about same-region transfer costs (https://github.com/supabase/supabase/discussions/3403) to not distract from the replication focus of this issue, but I wanted to leave a comment here and link the two ideas, because bring-your-own-bucket seems to be a good solution for both.
Yes, status please? Is it coming (soon), or should we find other workarounds?
