cloud_storage: Extend archival_metadata_stm

[Lazin]

trafficstars

Cover letter

This PR modifies archival_metadata_stm and partition_manifest to support retention and compaction. A collection of replaced segment is added to the manifest. The segment goes to this collection if it's replaced by the re-uploaded compacted segment. These segments are used by archival_metadata_stm to provide the backlog for data removal.

The archival_metadata_stm is extended to have the collection of replaced segments and start offset (both are stored explicitly in the snapshot).

Two new commands are added:

• update_start_offset_cmd used by retention
• cleanup_metadata_cmd used by the GC process that actually removes data from S3

The segment_meta is extended to store extra fields:

• segment_term - the term during which the segment was generated
• delta_offset_end - delta offset of the last record of the segment
• version is a version of the segment_meta struct

The version is used to generate segment names differently in the cloud. If the version has default value (segment was added by the older version) redpanda will use current segment name <offset>-<term>-v1.log.<archiver-term> but if it's set to v2 it will generate different name that contains a committed offset. This is needed to avoid situation when compacted segment overwrites the existing compacted or non-compacted segment in S3 bucket.

Fixes #6373

Backport Required

• [x] not a bug fix
• [ ] issue does not exist in previous branches
• [ ] papercut/not impactful enough to backport
• [ ] v22.2.x
• [ ] v22.1.x
• [ ] v21.11.x

UX changes

• none

Release notes

• none

Features

• none

Improvements

• none

[mmedenjak]

I assume this fixes https://github.com/redpanda-data/redpanda/issues/6373?

[Lazin]

I assume this fixes #6373?

yes

[emaxerrno]

i assume v1,v2 mixed partitions will always be allowed in case of a rolling upgrade

[Lazin]

i assume v1,v2 mixed partitions will always be allowed in case of a rolling upgrade

yes, this is exactly the case

[jcsp]

LGTM with the exception of the comment in partition_manifest::difference (https://github.com/redpanda-data/redpanda/pull/6387#discussion_r992318028)

Needs a rebase for the overlapping changes to partition_manifest_test.cc from my timequery PR.

[jcsp]

This fails to build

[jcsp]

Force push: this couldn't merge because it had a merge conflict with dev, so while rebasing I took the opportunity to:

• pull in the commit that fixes read replicas
• drop the commit that disables read replica tests
• update the SIPartitionMovementTest change to disable both variants of the test, as anything with mixed clusters will occasionally fail, and modify the commit message to reference a ticket https://github.com/redpanda-data/redpanda/issues/6837

If that runs through CI cleanly, that should position us to merge tomorrow morning.

[jcsp]

Push: add a commit to fix unit tests that were broken by addition of insync_offset to manifest.