Object Storage - asynchronous replication

[Izaia64]

As a user I want to be able to replicate object storage between regions I want to be able to consume these features through standard APIs / OVHcloud Customer panel Replication between 2 buckets, asynchronous replication (aprox. 15 minutes)

We will enable automatic and asynchronous replication of objects contained inside a source bucket to one or more destination buckets. Those destination buckets can be in one single region or dispatched throughout several regions according to your needs.

Async replication will allow customers to:

• make exact copies of objects (all metadata will also be replicated)
• address synchronization of data between several teams use cases
• transition copies of your data into other storage classes to optimize costs
• augment high resiliency by having your data replicated across multiple regions

What will be provided is a S3 compatible REST API to put, delete, download a replication configuration file that allows for definition of :

• destination buckets
• filters to select which objects to replicate
• rule priorities

The following configuration elements will be supported:

• replication to multiple destination buckets (including delete operation)
• replication to buckets located across multiple regions

The following configuration will NOT be supported for now:

• replication to buckets owned by a different account (destination buckets are not owned by the same account that owns the source bucket)
• transition copies to COLD STORAGE (deep archive) class
• transition to other storage classes
• retrieval of replication metrics (Nb of objects pending replication, Bytes pending replication, Operations failed replication, Replication latency (delta between buckets), Thoughput)
• replication of existing objects i.e objects that were present before replication was applied

[Izaia64]

This feature is in development and targeted for Q2 2023 (April/May)

[loliee]

This feature will not be compliant with SSE-C encrypted objects right ? It will be compliant with SSE-S3 encrypted objects ?

Thanks

[lason-ovh]

This feature will not be compliant with SSE-C encrypted objects right ? It will be compliant with SSE-S3 encrypted objects ?

Thanks

Hi, this feature will not be compliant with SSE-C nor SSE-S3 for now:

• for SSE-C, additional work will be needed so in the first iteration, it will not be supported yet
• SSE-S3 (and SSE-KMS) is a feature not available yet

Hope it helps

[lenglet-k]

Hi, have you any news for this feature ?

[lason-ovh]

Hi, have you any news for this feature ?

Hi, as previously stated, this feature is due in Q3 2023

[loliee]

Hi, is this feature available in an alpha or beta version ? Thanks

[biapar]

Ready?

[JustDoItSascha]

Hi, have you any news for this feature ?

Hi, as previously stated, this feature is due in Q3 2023

I love how you made Q2 from @Izaia64 to Q3. And now we have Q4 and there is no update :(

[agoude]

Hi @JustDoItSascha, well thanks for your patience and highlighting the lighter side of the situation ;) The team has been pushing hard rebuilding the feature to make it better and ensuring the performance/resilience levels we wanna reach: the main reason behind those different shifts and we are sorry for such delays!

The wait has been longer than expected and we're working to make it worth your wait. Now, I can only say we never been so closed to deploy it and i'll keep you posted with a solid and hopefully final ETA 🙏!

[JustDoItSascha]

Ok cool, thank you :)

[mahmoudaljan]

@agoude Thank you for the hard work. We are eager to get our hands on the feature. I know that for us, this will make our lives much easier. Don't hesitate to share an ETA if you have any.

[agoude]

@agoude Thank you for the hard work. We are eager to get our hands on the feature. I know that for us, this will make our lives much easier. Don't hesitate to share an ETA if you have any.

hey @mahmoudaljan we all agree on this feature importance, no doubt! As I said earlier, we will shortly keep you posted with a solid ETA! thanks!

[lason-ovh]

Hi everyone, I wanted to ask: since the replication applies to only new objects i.e objects that are added AFTER the replication configuration is uploaded, we were wondering the importance of replicating the objects that already exist in the bucket.

• on a scale of 1 (not important) to 5 (mandatory feature), what is for you the importance of providing a tool/mechanism to automatically replicate existing objects?
• on a scale of 1 (I don't mind) to 5 (I'd hate to), should we provide that tool, do you mind executing a (very) few command lines?

[biapar]

+4 +4 plus by control panel.

[mahmoudaljan]

@lason-ovh

• 5 : You must assume that every client who needs this feature will look for ways to achieve this, better to deal with it on your own terms (bandwidth...) and limit complaints and questions due to client mishaps while trying to achieve this on their own.

• 2 : Since this will only be a procedure that will be executed once, no one will lack motivation to make it work at the beginning even if it requires a multi step procedure with command lines. Also, please do provide an API call if possible.

[131]

• 5
• 2 // do whatever you want as long as terraform can apply it

[lenglet-k]

• 5
• 2

[JustDoItSascha]

• 5
• 5

[fredleger]

@lason-ovh

• 5 : You must assume that every client who needs this feature will look for ways to achieve this, better to deal with it on your own terms (bandwidth...) and limit complaints and questions due to client mishaps while trying to achieve this on their own.
• 2 : Since this will only be a procedure that will be executed once, no one will lack motivation to make it work at the beginning even if it requires a multi step procedure with command lines. Also, please do provide an API call if possible.

• 5
• 2 (as long as we can automate it in some way like script or jobs in our own infra)

[mahmoudaljan]

@agoude A little humor doesn't hurt.

https://giphy.com/gifs/fallontonight-jimmy-fallon-nbc-xUA7bcNrD7wxpIRgTS

[agoude]

@agoude A little humor doesn't hurt.

https://giphy.com/gifs/fallontonight-jimmy-fallon-nbc-xUA7bcNrD7wxpIRgTS

You're right and thanks for this!

[lason-ovh]

It's finally happening!!!! 😮 😮 The long awaited ASYNCHRONOUS REPLICATION will be very soon open in BETA (ETA = W14). You can already suscribe via the following link to be among our early users: https://survey.ovh.com/index.php/192786 Once the feature is activated, we will let you know. As always, we are keen on receiving your feedbacks, so do not hesitate! 😉

[lason-ovh]

Hi all, the Asynchronous Replication feature is now officially in beta!! 🥳 For those who signed up, you can now replicate your objects between buckets in the same region or in different regions. Check the official doc here: https://help.ovhcloud.com/csm/en-ie-public-cloud-storage-s3-asynchronous-replication-buckets?id=kb_article_view&sysparm_article=KB0062415 If you have questions and/or need help on this feature, do not hesitate to post a message on our Discord chan dedicated to S3

[martonivan]

Do you already have an estimated schedule about when the feature is expected to reach the 1.0 production phase?