pageserver: eviction for secondary mode tenants

[jcsp]

Follows #6123

Closes: #evictionhttps://github.com/neondatabase/neon/issues/5342

The approach here is to avoid using Layer from secondary tenants, and instead make the eviction types (e.g. EvictionCandidate) have a variant that carries a Layer for attached tenants, and a different variant for secondary tenants.

Other changes:

• EvictionCandidate no longer carries a Timeline: this was only used for providing a witness reference to remote timeline client.
• The types for returning eviction candidates are all in disk_usage_eviction_task.rs now, whereas some of them were in timeline.rs before.
• The EvictionCandidate type replaces LocalLayerInfoForDiskUsageEviction type, which was basically the same thing.

Problem

Summary of changes

Checklist before requesting a review

• [ ] I have performed a self-review of my code.
• [ ] If it is a core feature, I have added thorough tests.
• [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• [ ] Do not forget to reformat commit message to not include the above checklist

[github-actions[bot]]

2262 tests run: 2177 passed, 0 failed, 85 skipped (full report)

---

Flaky tests (3)

Postgres 15

• test_pg_regress: debug
• test_tenant_delete_races_timeline_creation: debug

Postgres 14

• test_statvfs_pressure_usage: debug

Code coverage (full report)

• functions: 54.5% (10235 of 18793 functions)
• lines: 81.5% (58908 of 72311 lines)

---

_{The comment gets automatically updated with the latest test results
12db7523caaf195192ed56009cf12db96c1c426c at 2024-01-16T10:23:47.435Z :recycle:}