`group.initial.rebalance.delay.ms` causes consumed messages' offset incorrect

[bennfocus]

Description

When I set "group.initial.rebalance.delay.ms=10000" on broker version "3.1.0", the messages polled by method consumer.poll(timeout=1.0) sometimes with from very high number offset() (such as 4272995135), even though the maximum offset on the partition was 35.

How to reproduce

set group.initial.rebalance.delay.ms=10000 on borker create an empty topic insert some message to the topic start 3 consumers on the topic call consumer.poll() one partition's offset may start from very high number

Checklist

Please provide the following information:

• [ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): ('1.7.0', 17235968)
• [ ] Apache Kafka broker version: 3.1.0
• [ ] Client configuration: {"enable.auto.commit": "true" "auto.commit.interval.ms": "10" "enable.auto.offset.store": "false" "auto.offset.reset": "earliest"}
• [ ] Operating system: Mac
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[bennfocus]

Weird, when I change group.initial.rebalance.delay.ms to 0, the issue disappeared. Then I changed it back to 10000, can not reproduce the issue again.

[bennfocus]

When I do stress test with producing lots of msgs and consuming them, the issue happened again. Basically the consumer got a message with very a high offset number message, don't know how it happened.

[mhowlett]

there was at least one PR merged for librdkafka 1.9.0 related to resolving incorrect offset behavior. it was not this issue specifically, but it could be relevant. can you try 1.9.0-RC7? (may need to build librdkafka, not sure where we're at with that).