[NEW] Atomic slot migration HLD

[PingXie]

trafficstars

Yet another gem that we should seriously consider for our new project. The original proposal (and the thread) is too long so I am going to leave it at https://github.com/redis/redis/issues/10933.

Note that this feature will be orthogonal to how cluster topology management is done.

[PingXie]

@madolson and I discussed another option (in the context of atomic slot migration) which essentially implements slot-level replication. Here is the high level flow

1. source (parent) forks a child with a set of slots to migrate
2. child streams slots to be migrated in the RDB format (same as replication)
3. target needs to support no-blocking load of the RDB stream (think of co-routines)
4. source (parent) starts capturing updates in the migrating slots right after forks
5. child completes streaming
6. source (parent) pauses clients writing to these slots
7. source replicates captured updates to target
8. source starts the slot ownership transfer process (depending on cluster v1 vs v2, we could take different paths)
9. source unblocks paused clients with -MOVED <target>

any failure on the target before completing step 8 would abort the migrating process on the source.

[zuiderkwast]

I have an implementation. Didn't write it down yet, but I recorded a song which explains it all. Please review. https://suno.com/song/b06c2a5b-3760-4916-9f56-eb3fe66f24e2

[PingXie]

I have an implementation. Didn't write it down yet, but I recorded a song which explains it all. Please review. https://suno.com/song/b06c2a5b-3760-4916-9f56-eb3fe66f24e2

Are you referring to https://github.com/valkey-io/valkey/pull/298? I will take a look next but I don't see it solving this issue?

[zuiderkwast]

@PingXie The song is about atomic slot migration. #298 is about subscribing to slot migrations. Not the same.

[PingXie]

@PingXie The song is about atomic slot migration.

Ship-it