redpanda-data
Kminion consumer group info gathering incompatible with Kafka 4.1.0
Running Kminion 2.2.14, connecting to an Apache Kafka cluster that was recently upgraded to version 4.1.0. Since that upgrade, Kminion isn't able to retrieve consumer group info any more. For each attempt to read consumer offsets, the following lines are logged:
{"level":"warn","ts":"2025-09-24T06:32:18.035Z","logger":"main.minion_service","msg":"failed to decode offset commit value","topic":"__consumer_offsets","partition_id":21,"offset":2259385709,"error":"response did not contain enough data to be valid"} {"level":"warn","ts":"2025-09-24T06:32:18.035Z","logger":"main.minion_service","msg":"failed to decode offset record","error":"failed to decode offset commit value: response did not contain enough data to be valid"}
This leads to no consumer lag, offset commit and related metrics being reported any more, rendering Kminion unusable for monitoring consumer groups.
Kafka's kafka-consumer-groups cli tool can read the offsets fine on version 4.1.0, but either it has been updated to whatever changes have been made to consumer offsets topic or it's using an internal API instead of relying on reading consumer offsets directly.
Is this issue already known? Have other come across this?
I haven't tested it with 4.1 yet, but they may have updated the format. You can use the different method to collect consumer group offsets (via the Kafka APi rather than reading from the topic) though.
Thanks for the hint regarding the possibility to use Kafka's admin api. I had not yet recognized that this is possible and for some reasons the Kminion in question had offsetsTopic configured as source scrapeMode.
Alright, so I tracked down the issue to an incompatibility in franz-go. This was fixed in the most recent release of franz-go, mainly kmsg v1.12.0.
I have created PR #317 to update franz-go in Kminion so it is fully compatible in offsetsTopic scrape mode with the benefits it brings.
With the changes being merged, is it possible to release a new version of Kminion please?
@schustersv Yes we should tag a new release. FWIW new docker images are built for every push against master: https://hub.docker.com/r/redpandadata/kminion/tags