Offset is getting reset to earliest after getting an "invalid offset" error

[shakhar]

trafficstars

• Version of Ruby: 2.5.3
• Version of Kafka: 3.0
• Version of ruby-kafka: 1.4

After upgrading our Kafka version to 3.0, we started to get the following error from our consumers: Invalid offset 238887998 for ${topic}, resetting to default offset

This caused our production consumers' offset to reset to earliest only on one partition and started consuming all our messages again from the start, what cause a huge lag.

We believe this issue cause because of a partition leader replacement that is not updated with the latest offset and when trying to fetching messages of the latest offset we get the above error.

We investigated the issue and saw that when getting the above error here, the offset is reset to default_offset that is default to earliest.

We also saw that if we will configure start_from_beginning to false here it may help prevent from it to rest the offset to earliest, but we are not sure if by that we can loose messages.

The behavior we expecting when getting this error is to reset the offset to the last processed offset of the consumer.

I'll be glad to hear your opinion about it, and If you can provide us a way to prevent this case from happening again 🙏🏽