[Issue 13559] Offload compacted ledger into tiered storage

[qq619618919]

Master Issue: #13559

Motivation

If topic is configured to offload, automatically offload compacted ledger into tiered storage.

Modifications

Offload compacted ledger into tiered storage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API: (yes / no)
• The schema: (yes / no / don't know)
• The default values of configurations: (yes / no)
• The wire protocol: (yes / no)
• The rest endpoints: (yes / no)
• The admin cli options: (yes / no)
• Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

• [x] doc-required

  (If you need help on updating docs, create a doc issue)

• [ ] no-need-doc

  (Please explain why)

• [ ] doc

  (If this PR contains doc changes)

[github-actions[bot]]

@qq619618919:Thanks for your contribution. For this PR, do we need to update docs? (The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

[Anonymitaet]

@qq619618919 when submitting a PR, can you provide doc-related info (tick the box) in the PR description? So that Bot can recognize and label your PR correspondingly. Or else Bot labels your PR with doc-info-missing.

Instructions of doc labels: https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0/edit#bookmark=id.5d66olk7l4oz

[github-actions[bot]]

@qq619618919:Thanks for providing doc info!

[codelipenghui]

@qq619618919 For a new feature, it's better to follow the PIP process https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-%28PIP%29

[qq619618919]

Changes:

1. Add compactedTopic offload code in CompactorSubscription. If the topic is successfully compacted to bookkeeper, it will continue to check whether there is an offload ledger on this topic. If there is, it will offload the compacted topic ledger data from bookkeeper to tiered storage, and save the uuid of the successful offload attempt to MetaStore.
2. Change LedgerHandle in CompactedTopicContext to ReadHandle of it‘s super interface, which ensures that bookkeeper and tiered storage share ReadHandle. And add ReadHandle to read data related logic.
3. Add an OffloadCompactedTopicImpl class that extends CompactedTopicImpl, Indicates that the CompactedTopic is in the offload state. If you need to read data from OffloadCompactedTopicImpl, it will use the ReadHandle of tiered storage to read. At the same time, when the topic is initialized, the attributes of the compaction subscription metadata will be detected. If there is an offloaded uuid, the OffloadCompactedTopicImpl will be constructed, which ensures that the data can still be read from the tiered storage.

[github-actions[bot]]

The pr had no activity for 30 days, mark with Stale label.

[github-actions[bot]]

The pr had no activity for 30 days, mark with Stale label.