fix(pageserver): make image split layer writer finish atomic

[skyzh]

Problem

Part of https://github.com/neondatabase/neon/issues/8836

Summary of changes

This pull request makes the image layer split writer atomic when finishing the layers. All the produced layers either finish at the same time, or discard at the same time. Note that this does not prevent atomicity when crash, but anyways, it will be cleaned up on pageserver restart.

Checklist before requesting a review

• [x] I have performed a self-review of my code.
• [ ] If it is a core feature, I have added thorough tests.
• [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• [ ] Do not forget to reformat commit message to not include the above checklist

[github-actions[bot]]

5229 tests run: 5015 passed, 0 failed, 214 skipped (full report)

---

Flaky tests (1)

Postgres 15

• test_isolation[None]: release-arm64

Code coverage* (full report)

• functions: 31.2% (7568 of 24249 functions)
• lines: 48.8% (59936 of 122767 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
cc8f2452e6a45a3438738591e86b0313c6735ef5 at 2024-10-21T14:34:45.764Z :recycle:}

[skyzh]

comments resolved and ready for review again :)

[skyzh]

ready for review; the new commit also fixes the problem of image layer writer that it does not attempt to remove the temporary file if anything goes wrong

[skyzh]

@arpad-m do you mind have a final look of this PR since Christian is busy + this patch has been there for too long? Thanks a lot :)