feat(pageserver): support retain_lsn in bottommost gc-compaction

[skyzh]

Problem

part of https://github.com/neondatabase/neon/issues/8002

Summary of changes

The main thing in this pull request is the new generate_key_retention function. It decides which deltas to retain and generate images for a given key based on its history + retain_lsn + horizon.

Checklist before requesting a review

• [x] I have performed a self-review of my code.
• [ ] If it is a core feature, I have added thorough tests.
• [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• [ ] Do not forget to reformat commit message to not include the above checklist

[github-actions[bot]]

3126 tests run: 3005 passed, 0 failed, 121 skipped (full report)

---

Code coverage* (full report)

• functions: 32.6% (7019 of 21509 functions)
• lines: 50.2% (55718 of 111048 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
24fb4d551d6f48c34a90272f1d2d6b03d58ea6cb at 2024-07-23T19:24:25.817Z :recycle:}

[skyzh]

added tests for the core generate_key_retention function and ready for review :)

[skyzh]

added timeline e2e unit tests

[skyzh]

So, apart from the added tests, changes since my last review are that we now use lowest_retain_lsn instead of gc_cutoff

The lowest image layer is created over lowest_retain_lsn instead of gc_cutoff. gc_cutoff can be considered as a retain_lsn.

The current code, afaict, will create multiple deltas with LSN_range.start == lowest_retain_lsn.

Yes, for now, the bottom-most compaction always produce delta layers of the LSN range from lowest_retain_lsn to the GC horizon. We could further split it over the LSN as an optimization in the future.

[problame]

Yes, for now, the bottom-most compaction always produce delta layers of the LSN range from lowest_retain_lsn to the GC horizon. We could further split it over the LSN as an optimization in the future.

Are we not respecting the target file size yet?

[skyzh]

Are we not respecting the target file size yet?

Yes, not for now, this is tracked in the epic.