Kafka consumer not reset offset when CURRENT-OFFSET greater than LOG-END-OFFSET

[ducnt114]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
v1.26.4	kafka_2.12-2.1.0	1.12.10

Configuration

What configuration values are you using for Sarama and Kafka?

config := sarama.NewConfig()
config.Version = sarama.V1_1_0_0
config.Consumer.Return.Errors = true
config.Consumer.Offsets.Initial = sarama.OffsetOldest

Logs

Problem Description

Kafka consumer not reset offset when CURRENT-OFFSET greater than LOG-END-OFFSET

Scenario:

• Publish 10 messages to topic topic-test-1
• Start a consumer group cg-id-10 and consume 10 message
• Stop kafka server, delete data folder of topic topic-test-1 and then restart kafka server. Now topic topic-test-1 is empty.
• Push 6 messsages to topic topic-test-1.
• Consumer cg-id-10 continue to consume 6 new messages, but it does not commit offset to the server:

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cg-id-10 --describe

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                 HOST            CLIENT-ID
topic-test-1    0          10              6               -4              sarama-ffbd327e-db40-4443-855b-9ed68dcc57ad /10.124.52.185  sarama

If I restart consumer, it consume message from beginning all the time, instead of starting from offset 6th as needed.

I think that consumer need to set CURRENT-OFFSET to value 6 after consume all new messages in this case.

[khong07]

To add more information, with Java client, we don't have this problem. The consumer automatically reset the current offset to log end offset depends on offset.auto.reset strategy

[khong07]

Could someone take a look in this issue @burke @bai thanks

[ghost]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the master branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[khong07]

Hello, Any body could take the look into this issue?

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[dnwe]

To add more information, with Java client, we don't have this problem. The consumer automatically reset the current offset to log end offset depends on offset.auto.reset strategy

Hmm. So Java client ignores committed offset if it considers it invalid?

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.