Epic: improved compaction and compaction algorithms framework

[arpad-m]

Motivation

The algorithms in the pageserver's compaction component aren't perfect and can be improved.

DoD

We have implemented pluggable compaction support, see #5234. We also have implemented some (default off) algorithms and done some initial experiments with them, but commitment to rolling out any larger algorithm changes is optional, not required. The goal is to have the pluggable support merged before we go GA.

Implementation ideas

### Tasks
- [ ] #5234
- [ ] https://github.com/neondatabase/neon/pull/6830
- [ ] Making sure the files aren't too large: https://github.com/neondatabase/neon/pull/5234#discussion_r1333986725

Other related tasks and Epics

• Earlier PR #4539

[jcsp]

This week:

• Get the existing PR green for merge.

[arpad-m]

Last week: got PR rebased, polished, reviewed, and addressed reviews. This week: merge the PR, work on analyzing the new compaction's characteristics, reading the literature, and work on followups.

[problame]

• read more literature
• follow-ups
  • PR this week, need to confer with Heikki about some aspects of the code

[jcsp]

This week:

• Compare old/new on gc_feedback (staircasing) test case.
• Fix the edge case where too many deltas on one key can result in oversized layer files.

Deferring work to make all the tests work with new style compaction (as far as we know these are test assumption issues rather than issues with new compaction) -- the near term goal is to reach "no known issues" status with new compaction.

[problame]

Redefined this issue as the work that has been done in Q1:

~~Epic: improved compaction and compaction algorithms framework~~ Epic: import compaction framework & tiered compaction code

And moved the uncompleted items into section

Future Work / Punted into Q2 https://github.com/neondatabase/neon/issues/7554