neon icon indicating copy to clipboard operation
neon copied to clipboard

WAL filtering in safekeeper for sharding

Open petuhovskiy opened this issue 1 year ago • 0 comments

Arthur's prototype from January: https://github.com/neondatabase/neon/tree/sk-sharding-stream

Precursors

We may start by refactoring pageserver WAL ingest code to make decoding WAL records more independent of Timeline.

### Tasks
- [ ] https://github.com/neondatabase/neon/pull/8754
- [ ] https://github.com/neondatabase/neon/pull/8591
- [ ] https://github.com/neondatabase/neon/pull/8621
- [ ] Defer applying metadata changes until commit() time in DataDirModification (stop carrying a Timeline ref)
- [ ] Write GC cutoff to S3 from Shard 0, and stop storing SLRU data on other shards.

Implement splitting on safekeeper

### Tasks
- [ ] Transmit ShardIndex to safekeeper when subscribing for WAL
- [ ] Change WAL send protocol to send a decoded DataDirModification instead of a raw WAL record
- [ ] Filter modifications on the safekeeper to only include pages relevant to the shard who is subscribed

Optimizations

### Tasks
- [ ] Implement a per-Timeline cursor to avoid reading from disk & decoding repeatedly for each shard

petuhovskiy avatar Jan 12 '24 11:01 petuhovskiy