KAFKA-14412: Use shared cache for Task offset sums
Instead of reading Task state offsets for non-open Tasks from the
.checkpoint file, we now maintain an in-memory cache of the latest
changelog offsets for every Task on the instance.
On start-up, this cache is seeded with the changelog offsets for every on-disk StateStore. Running Active and Standby Tasks then update this cache on every checkpoint to ensure it always reflects the offsets on-disk.
This breaks the tight coupling between TaskManager and .checkpoint
files, which will enable us to remove .checkpoint files in a later
commit as part of KIP-1035.
@ableegoldman @mjsax @cadonna I had some time to pick this up, so I've rebased it against trunk and re-submitted it.
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.