neon
neon copied to clipboard
WAL filtering in safekeeper for sharding
Arthur's prototype from January: https://github.com/neondatabase/neon/tree/sk-sharding-stream
Precursors
We may start by refactoring pageserver WAL ingest code to make decoding WAL records more independent of Timeline.
### Tasks
- [ ] https://github.com/neondatabase/neon/pull/8754
- [ ] https://github.com/neondatabase/neon/pull/8591
- [ ] https://github.com/neondatabase/neon/pull/8621
- [ ] Defer applying metadata changes until commit() time in DataDirModification (stop carrying a Timeline ref)
- [ ] Write GC cutoff to S3 from Shard 0, and stop storing SLRU data on other shards.
Implement splitting on safekeeper
### Tasks
- [ ] Transmit ShardIndex to safekeeper when subscribing for WAL
- [ ] Change WAL send protocol to send a decoded DataDirModification instead of a raw WAL record
- [ ] Filter modifications on the safekeeper to only include pages relevant to the shard who is subscribed
Optimizations
### Tasks
- [ ] Implement a per-Timeline cursor to avoid reading from disk & decoding repeatedly for each shard