kafka
kafka copied to clipboard
GroupCoordinatorNotAvailable Issue
I'm trying use docker-compose to bring up both my group consumer and a docker container running kafka. There is an issue with timing where some times the consumer init code is getting called before kafka is done its startup procedure. I would like to be able to retry the group consumer init after kafka has finished starting.
What I see is [Error: Failed to join the group: GroupCoordinatorNotAvailable]
. When I notice that error, I attempt to instantiate a new GroupConsumer with the same config, but then see the message Sync group: unknown topic: null
I know I'm configuring the consumer correctly, because when the timing works out and kafka is fully started before the init code runs, the GroupConsumer init completes.
It looks like in this library's client.updateGroupCoordinator function, it may be getting confused after a GroupCoordinatorNotAvailable error. Not exactly sure, though.
I tend to write a custom run script in bash (or just sh) for most of my applications with a helper function such as
function testConnection {
local tries=0
local maxTries=20
local sleepSeconds=1
echo "Testing connection"
while ! exec 6<>/dev/$3/$1/$2; do
((tries++))
echo "Connection failed ($tries/$maxTries)"
if [ "$tries" == "$maxTries" ]; then
return 1
fi
sleep $sleepSeconds
done
exec 6>&-
exec 6<&-
echo "Connection was successful"
return 0
}
You simply use it like
if ! testConnection "<ip/hostname>" "<port>" "<tcp/udp>"; then
exit 1
fi
I am not entirely sure if kafka listens for connections prior to being completely ready, but I guess it shouldn't.
Now, this will make the script fail after 20 tries, this is where I use restart: always
to simply try again, and again.
You can of course do some clever application code instead of going to bash for help, but personally, I don't always use node, so I love having a generic solution.