kafka icon indicating copy to clipboard operation
kafka copied to clipboard

KAFKA-14412: Use shared cache for Task offset sums

Open nicktelford opened this issue 1 month ago • 3 comments

Instead of reading Task state offsets for non-open Tasks from the .checkpoint file, we now maintain an in-memory cache of the latest changelog offsets for every Task on the instance.

On start-up, this cache is seeded with the changelog offsets for every on-disk StateStore. Running Active and Standby Tasks then update this cache on every checkpoint to ensure it always reflects the offsets on-disk.

This breaks the tight coupling between TaskManager and .checkpoint files, which will enable us to remove .checkpoint files in a later commit as part of KIP-1035.

nicktelford avatar Nov 21 '25 18:11 nicktelford

@ableegoldman @mjsax @cadonna I had some time to pick this up, so I've rebased it against trunk and re-submitted it.

nicktelford avatar Nov 21 '25 18:11 nicktelford

A label of 'needs-attention' was automatically added to this PR in order to raise the attention of the committers. Once this issue has been triaged, the triage label should be removed to prevent this automation from happening again.

github-actions[bot] avatar Nov 29 '25 03:11 github-actions[bot]

A label of 'needs-attention' was automatically added to this PR in order to raise the attention of the committers. Once this issue has been triaged, the triage label should be removed to prevent this automation from happening again.

github-actions[bot] avatar Dec 08 '25 03:12 github-actions[bot]