mimir icon indicating copy to clipboard operation
mimir copied to clipboard

Clarify logic around partition consumed offset

Open pracucci opened this issue 4 months ago • 0 comments

What this PR does

Currently, the logic around the last consumed offset which is committed to the Kafka consumer group is a bit convoluted: we don't commit the last consumed offset, but the last consumed offset + 1, which is the offset at which we should start replaying at the next startup.

In this PR I'm clarifying the logic, committing the last consumed offset. Then, when we read it back at PartitionReader startup, if we found the committed last consumed offset we'll start replaying from offset + 1.

The logic in this PR is expected to be unchanged, except for the metric cortex_ingest_storage_reader_last_committed_offset which is now fixed and track the last consumed offset, instead of the last consumed offset + 1.

Which issue(s) this PR fixes or relates to

N/A

Checklist

  • [ ] Tests updated.
  • [ ] Documentation added.
  • [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • [ ] about-versioning.md updated with experimental features.

pracucci avatar Mar 26 '24 13:03 pracucci