[General question] Restoring huge buckets (>20TB)

[MattiDeGrauwe]

Hi all

I've been trying to restore an entire bucket to a specific state (eg: Monday 08-26-2021 23:59:59 +2). I tried to restore the entire bucket to this state using the following command:

/usr/local/bin/s3-pit-restore -b [bucketname] -d [bucketname] -f "08-26-2021 00:00:00 +2" -t "08-26-2021 23:59:59 +2" --max-workers 100 -v

I noticed that his command copies every S3 object to a new version. I'm trying to only copy objects that are modified after 26 August in this case, is this possible?

What exactly am I doing wrong in my case?

Thanks in advance.

[angeloc]

Hi all

I've been trying to restore an entire bucket to a specific state (eg: Monday 08-26-2021 23:59:59 +2). I tried to restore the entire bucket to this state using the following command:

/usr/local/bin/s3-pit-restore -b [bucketname] -d [bucketname] -f "08-26-2021 00:00:00 +2" -t "08-26-2021 23:59:59 +2" --max-workers 100 -v

I noticed that his command copies every S3 object to a new version.

What does this mean? You are downloading a copy of the objects locally

I'm trying to only copy objects that are modified after 26 August in this case, is this possible?

Your to/from restricts the objects to be downloaded to one day only, not after the 26 of August

What exactly am I doing wrong in my case?

Thanks in advance.

[MattiDeGrauwe]

Hi Angeloc,

Thanks for your swift reply. It seems like I pasted the wrong command, wherefor my apologies.

When executing the following command: /usr/local/bin/s3-pit-restore -b s3-tvhbel-dam-data-prd-us-east-2-oxkux-mirror -B s3-tvhbel-dam-data-prd-us-east-2-oxkux-mirror -t "08-26-2021 23:59:59 +2" --max-workers 100 -v

It restores all objects changed before 26 August apparently, also the ones that weren't modified after 26 August. Which command should I execute to restore the bucket to the state it was at 26 August?

Warm regards

[MattiDeGrauwe]

Hello Angeloc

I think that what I'm trying to achieve is exactly what is requested in https://github.com/angeloc/s3-pit-restore/pull/5 I think my command should be correct eventually, can you confirm this?

Thanks in advance.