sarama icon indicating copy to clipboard operation
sarama copied to clipboard
verified publisher icon IBM

kafka retains old generations of consumer group members.

Open boralt opened this issue 3 years ago • 2 comments

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama Kafka Go
1.27.2 2.13-2.6.2 1.17.3
Configuration

What configuration values are you using for Sarama and Kafka?

Logs
[2022-02-23 03:01:05,743] INFO Static member MemberMetadata(memberId=sarama-7d51bee2-b1f6-4f3c-b9c8-4b28a15e11ab, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-7d51bee2-b1f6-4f3c-b9c8-4b28a15e11ab at generation 1. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,744] INFO Static member MemberMetadata(memberId=sarama-a19dcd7d-0e4b-452e-8bf4-f6d6c3389125, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-a19dcd7d-0e4b-452e-8bf4-f6d6c3389125 at generation 3. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,744] INFO Static member MemberMetadata(memberId=sarama-b25f5b68-6531-4b67-b146-3a5a6c4911d9, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-b25f5b68-6531-4b67-b146-3a5a6c4911d9 at generation 5. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,751] INFO Static member MemberMetadata(memberId=sarama-4b27bac7-6b44-498e-88de-9017d4859584, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b27bac7-6b44-498e-88de-9017d4859584 at generation 7. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,761] INFO Static member MemberMetadata(memberId=sarama-4b27bac7-6b44-498e-88de-9017d4859584, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b27bac7-6b44-498e-88de-9017d4859584 at generation 8. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,767] INFO Static member MemberMetadata(memberId=sarama-2790bd7d-5919-49d4-8601-f872931dab71, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-2790bd7d-5919-49d4-8601-f872931dab71 at generation 10. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,768] INFO Static member MemberMetadata(memberId=sarama-ede585d4-ba23-4ce9-82f6-3d108a522705, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-ede585d4-ba23-4ce9-82f6-3d108a522705 at generation 12. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,772] INFO Static member MemberMetadata(memberId=sarama-cb23ec41-9b52-4f4e-ba86-9ef20a69aeee, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-cb23ec41-9b52-4f4e-ba86-9ef20a69aeee at generation 14. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,779] INFO Static member MemberMetadata(memberId=sarama-c625e0e8-c38b-48c5-a56e-966c963b30f6, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-c625e0e8-c38b-48c5-a56e-966c963b30f6 at generation 16. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,779] INFO Static member MemberMetadata(memberId=sarama-d36e64ab-0706-4abf-a01c-f87ea96ba455, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-d36e64ab-0706-4abf-a01c-f87ea96ba455 at generation 18. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,783] INFO Static member MemberMetadata(memberId=sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21 at generation 20. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,798] INFO Static member MemberMetadata(memberId=sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21 at generation 21. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,799] INFO Static member MemberMetadata(memberId=sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21 at generation 22. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,799] INFO Static member MemberMetadata(memberId=sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21 at generation 23. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,801] INFO Static member MemberMetadata(memberId=sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21 at generation 24. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,801] INFO Static member MemberMetadata(memberId=sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-4b4819b2-943d-4922-bc83-ad5fafee7e21 at generation 25. (kafka.coordinator.group.GroupMetadata$)
[2022-02-23 03:01:05,802] INFO Static member MemberMetadata(memberId=sarama-71324229-a703-4695-a109-a712b8bb0647, groupInstanceId=Some(null), clientId=sarama, clientHost=/192.168.1.207, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(range), ).groupInstanceId of group test3 loaded with member id sarama-71324229-a703-4695-a109-a712b8bb0647 at generation 27. (kafka.coordinator.group.GroupMetadata$)
Problem Description

We have observed kafka brokers carry old generations of cg instance members as it shown in rebalance log. We don't see that with other clients. Not sure if that contributing to other problem we seen, but I wanted to get more clarity if this a real issue or not.

boralt avatar Feb 23 '22 03:02 boralt

@boralt some questions

  1. I'm not sure what you mean by "old" as the timestamps of these log lines are very close together, is the generationID just getting bumped due to consumer group instability as your clients join?
  2. do you have more of the broker log context?
  3. the client(s) listed all show as coming from the same 192.168.1.207 address — are you running multiple client instances from the same source IP or just one?

dnwe avatar Apr 14 '22 00:04 dnwe

@dnwe Here are some answers:

  1. We are running a single client instance, but in consumer group mode. No other clients ever join or leave this consumer group. We running this a cluster with 3 brokers. Tests includes injecting some network outages to one of the brokers which sometimes causing rebalancing applied to this one client. So these generations are really old corresponding to old rebalancing cycles, but they somehow never removed from metadata. This situation is totally reproducible, it happens every time network disruptions are applied to this configuration.
  2. I had megabytes of broker logs. Which type of log would be most usefull?
  3. Answered above.

Thanks

boralt avatar Apr 14 '22 01:04 boralt

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

github-actions[bot] avatar Aug 18 '23 08:08 github-actions[bot]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

github-actions[bot] avatar Nov 16 '23 10:11 github-actions[bot]