kafka
kafka copied to clipboard
Support for 0.10.x kafka version
Hi ,
I have been using no-kafka for 3-months now with kafka version 0.9 . A month ago, I upgraded to kafka 0.10.1.0, but most of our client-side code is already written using no-kafka, and as indicated in wiki, I thought no-kafka won't be compatible now, as it supports kafka 0.9.x . After kafka 0.10 upgrade , I have seen many times that consumers in consumer-group sleeps off even when above 2000 messages or so are in the queue unprocessed. Sometimes, it starts off automatically , while other times I have to restart the consumer docker containers.
I just wanted to ask whether the behaviour I am seeing is a result of version incompatibility? Is there any plan to add support for kafka 0.10.x, and if yes, when could be the expected time-frame here?
Thanks for giving a great software.
I don't think this behaviour has something to do with Kafka version upgrade. Kafka 0.10 added some new features but there were no core changes. I guess you should do some debugging both server side and client side, it might be some synchronous code running that blocks fetches, it might be network problems.
Hi Oleksiy,
This behaviour happened just 2-3 hours ago with me and I think blocking code is not a problem here, because of the following:
I have made a monitoring service that produces one record every 5 minutes in the kafka broker, and the no-kafka consumer , that is subscribed to the topic, is supposed to consume this message. Since only 1 record is produced , that too every 5 minutes, hence sometimes it takes even less than 0.5 seconds to consume, but sometimes, the consumer takes 5 minutes to consume the message, even when it is idle for last 5 minutes. I have timestamped the record consumption time and record production time in mongodb for every message , and have verified this behaviour.
Also , I have used async/await to make my code synchronous, and my memory and cpu are not clogged that might cause cpu scheduling or memory swapping .