[improve][pip] PIP-317: Add `bookkeeperDeleted` field to show whether a ledger is deleted from the bookie while using tiered storage

[dragonls]

Releted PR/issue: #20833

• [ ] doc
• [ ] doc-required
• [x] doc-not-needed
• [ ] doc-complete

[dragonls]

Should I start a vote for this PIP?

[zymap]

Yeah. Please go ahead.

[dragonls]

• Mailing List discussion thread: https://lists.apache.org/thread/9kjgg06oj384myxqls73dwn840thtrrc
• Mailing List voting thread: https://lists.apache.org/thread/5nfm3qkbwqylk1zp77zn9tfokhy212tb

[eolivelli]

It will be important to add tests about rolling back to the previous version of Pulsar and/or clusters with mixed versions of the brokers (during rolling upgrades). The old version should not complain about the new field added.

[dragonls]

It will be important to add tests about rolling back to the previous version of Pulsar and/or clusters with mixed versions of the brokers (during rolling upgrades). The old version should not complain about the new field added.

Do you mean adding test cases or updrading cluster including this feature?

The change of this feature is mainly to add a field bookkeeperDeleted in org.apache.pulsar.common.policies.data.ManagedLedgerInternalStats.LedgerInfo. The main scope of influence here is the result returned by the external interface for obtaining status, which will not affect the storage of ledger original data on zk.

This feature has been running on our online cluster for over half a year. No problems were found during the initial rolling upgrade process.

[hangc0276]

It will be important to add tests about rolling back to the previous version of Pulsar and/or clusters with mixed versions of the brokers (during rolling upgrades). The old version should not complain about the new field added.

Do you mean adding test cases or updrading cluster including this feature?

The change of this feature is mainly to add a field bookkeeperDeleted in org.apache.pulsar.common.policies.data.ManagedLedgerInternalStats.LedgerInfo. The main scope of influence here is the result returned by the external interface for obtaining status, which will not affect the storage of ledger original data on zk.

This feature has been running on our online cluster for over half a year. No problems were found during the initial rolling upgrade process.

@dragonls Thanks for driving this improvement. I have one concern about the compatibility the same with Enrico. The proposal adds one field in the ManagedLedgerInternalStats, which is located on the common package. Can we ensure compatibility when the server upgrades to the new version but the client stays on the old version? It happens in two cases:

• During upgrade and some broker in new version but others in old version
• Pulsar client in old version which doesn't has the new field.