rename S3 prefix on staging-ps2 to ps-staging-v2

[LizardWizzard]

Ideally it should be one prefix called e g v1 standing for the storage format version. But to accomplish that we'll need to rename prefixes on all pageservers.

Currently on staging there are two active pageservers, so to simplify that we can rename ps2 prefix to ps3 one to move only ps2 data into ps3 prefix.

Steps to do the ideal variant:

1. create v1 prefix
2. copy (can we move rather than copy?) ps2 and ps3 to v1
3. change configs in ps2 ps3 to point to v1 (they start uploading data that is present locally but missing in s3, remote_consistent _lsn can move backwards which is bad, but safekeepers upload wal to s3 so should be no problem)
4. check that nothing is present only in old prefix, if there is something missing from newer directory, copy again + restart pageservers so they can sync their internal state

Ideally we should have maintenance option for pageservers so projects are stopped and cannot start until pageserver they're connected to is on maintenance. @ololobus do we have that planned?

We'll need to perform the same dance on prod.

[LizardWizzard]

@zoete just to be sure, do we specify single prefix for pageserveers in new regions? To avoid repeating this on other regions...

UPD: Other concerns regarding prefix format?

[ololobus]

Ideally we should have maintenance option for pageservers so projects are stopped and cannot start until pageserver they're connected to is on maintenance. @/ololobus do we have that planned?

No, I periodically think about some maintenance mode for projects, but didn't think about storage nodes. We have active flag, setting it to false will prevent scheduling new tenants to it, but old one still can start. We can add another one, like maintenance, but what should we do with already running computes?

[LizardWizzard]

No, I periodically think about some maintenance mode for projects,

It will also work. We can set flag in projects based on pageserver they're connected to

but what should we do with already running computes?

Any other options other than shutting them down?

[LizardWizzard]

After discussions with Stas we came up with these two options for migration process with decreased downtime period.

Double sync approach.

1. Without shutting down pageserver, aws s3 sync <old prefix> <new prefix>
2. Shut down pageserver
3. Repeat sync
4. Restart pageserver with prefix switched to

Short downtime for final sync. Diff should be small. Has benefit of not deviating from standard operating modes. Because we currently do not test scenario when remote storage is disabled for some time and then reenabled.

Disable, copy enable

1. Disable remote storage, restart pageserver
2. aws s3 cp <old prefix> <new prefix>
3. aws s3 sync --dryrun to check that there is no difference
4. Restart pageserver with changed prefix and enabled remote storage.

This has the potential problem that we dont track pending deletions across restarts. E g if compaction deletes something then after restart this delete will be lost, and pageserver will download this data back during initial sync so it will end up having both compacted and non compacted version of the data at the same time which may cause unexpected issues.

I plan to write a test for that to see how it behaves in this case. Probably better to wait until I get the results before going further.

There is a third obvious variant with shutting down pageserver completely to do all manipulations as it is stated in the issue above. This looks safer to me, but extends downtime period.

Any other variants I missed? Concerns? Other thoughts? @kelvich @hlinnaka @SomeoneToIgnore

I plan to check selected variant on staging and stress env before applying it to prod

[LizardWizzard]

Done as part of account migration.