KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager

[aheev]

Reopen https://github.com/apache/kafka/pull/14136

• Rebase with latest trunk and account for new changes
• Replace TopicPartition with TopicIdPartition in StopPartition to handle a case where MetadataCache is updated first on removeTopic delta, so, during ReplicaManager.applyDelta, RLM can't find the topicId

PS: The race between LeaderAndISR and UpdateMetadata requests are no longer valid as, in KRaftMetadataCache is first updated before applying topicsDelta

JIRA: https://issues.apache.org/jira/browse/KAFKA-15038

Reviewers: @kamalcph

[aheev]

@divijvaidya @kamalcph could you please take a look at this?

[kamalcph]

@aheev

Thanks for the PR! Could you add a comment to understand why PR#14136 was abandoned and how this PR addresses it?

[aheev]

@aheev

Thanks for the PR! Could you add a comment to understand why PR#14136 was abandoned and how this PR addresses it?

updated

[github-actions[bot]]

A label of 'needs-attention' was automatically added to this PR in order to raise the attention of the committers. Once this issue has been triaged, the triage label should be removed to prevent this automation from happening again.

[aheev]

@kamalcph gentle ping for the review

[github-actions[bot]]

A label of 'needs-attention' was automatically added to this PR in order to raise the attention of the committers. Once this issue has been triaged, the triage label should be removed to prevent this automation from happening again.

[aheev]

will look at the test failures today

[github-actions[bot]]

A label of 'needs-attention' was automatically added to this PR in order to raise the attention of the committers. Once this issue has been triaged, the triage label should be removed to prevent this automation from happening again.

[aheev]

@kamalcph found a new issue. Since metadataCache is first updated before ReplicaManager.applyDelta, stopPartitions, inturn rlm.stopPartitions is not able to find topicId. Why can't we just use TopicIdPartition in StopPartition class :thinking:

[github-actions[bot]]

A label of 'needs-attention' was automatically added to this PR in order to raise the attention of the committers. Once this issue has been triaged, the triage label should be removed to prevent this automation from happening again.

[chia7712]

@aheev could you please rebase code to include CI changes?

[aheev]

Thanks for the patch!

Since metadataCache is first updated before ReplicaManager.applyDelta, stopPartitions, inturn rlm.stopPartitions is not able to find topicId. Why can't we just use TopicIdPartition in StopPartition class

yeah, I think topicId can be set in the StopPartition object to propagate the value to the RemoteLogManager. The deletedTopicIds is available in the TopicsDelta.

Replaced TopicPartition with TopicIdPartition in StopPartition

[github-actions[bot]]

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.