Epic: Read Latency — define metrics, define and achieve launch objectives

[stepashka]

To prepare for the work on this epic we need to..

• [ ] define and collect the Read Latency metrics
• [ ] collect the Read Latency metrics from both zenith and vanilla postgres for comparison and benchmark
• [ ] #1380
• [ ] log span and metrics overhead

Once we start collecting these measurements, we'd set a goal for them for the Technical Preview. Below are the ideas on what could help us achieve those goals.

Within page server:

• [ ] #1358
• [ ] #1339
• [ ] serde deserialization
• [ ] page cache lookups
• [ ] #733

Compute / pageserver interface:

• [ ] prefetching, zenithdb/postgres#133
• [ ] bulk reads (for seqscans) zenithdb/postgres#133
• [ ] caching in compute node OR RATHER deploy pageserver in compute node zenithdb/postgres#117
• [ ] #1362
• [ ] #1537
• [ ] #1522
• [x] #1919

[stepashka]

once we've merged the WAL redo speed changes, this goes back on ice

[knizhnik]

Actually I do not understand why we are so optimistic here. This wal redo optimization provides ~1o% percent improvement. Launching more than one wal-redo postgres is IMHO no so good idea, because provides small (10%) improvement for multiclient read-only workload and at my tests is shows some degrade of performance for single client workload. And it double s number of processes which also may have negative impact on overall resource consumption and performance.

So we have not found "silver bullet" and wal redo speed is still very slow. If working set doesn't fit in compute node workset and we have to retrieve pages from page server, then we are about 3-5 times slower than vanilla posgtres. Rewriting wal redo is rust is not a realistic solution right now and it still doesn't completely solve the problem (still page reconstruction is times slower than reading page image).

So if we can not speed up wal redo, then we should try to avoid wal redo. It can be done in many different ways:

1. Increase cache size at compute node (including local file cache)
2. Use large page cache at page server (may be also on disk). Materialized page eviction policy has to be adjusted.
3. Force generation of image layers so before page is requested is is already reconstructed.

None of this approaches were discussed (although we have some number of PRs in this directions) and looks like we are not going to try to do something before launch.