GroupCoordinatorNotAvailable Issue

[Mattenable]

I'm trying use docker-compose to bring up both my group consumer and a docker container running kafka. There is an issue with timing where some times the consumer init code is getting called before kafka is done its startup procedure. I would like to be able to retry the group consumer init after kafka has finished starting.

What I see is [Error: Failed to join the group: GroupCoordinatorNotAvailable]. When I notice that error, I attempt to instantiate a new GroupConsumer with the same config, but then see the message Sync group: unknown topic: null I know I'm configuring the consumer correctly, because when the timing works out and kafka is fully started before the init code runs, the GroupConsumer init completes.

It looks like in this library's client.updateGroupCoordinator function, it may be getting confused after a GroupCoordinatorNotAvailable error. Not exactly sure, though.

[Multiply]

I tend to write a custom run script in bash (or just sh) for most of my applications with a helper function such as

function testConnection {
    local tries=0
    local maxTries=20
    local sleepSeconds=1

    echo "Testing connection"
    while ! exec 6<>/dev/$3/$1/$2; do
        ((tries++))
        echo "Connection failed ($tries/$maxTries)"
        if [ "$tries" == "$maxTries" ]; then
            return 1
        fi

        sleep $sleepSeconds
    done
    exec 6>&-
    exec 6<&-

    echo "Connection was successful"
    return 0
}

You simply use it like

if ! testConnection "<ip/hostname>" "<port>" "<tcp/udp>"; then
    exit 1
fi

I am not entirely sure if kafka listens for connections prior to being completely ready, but I guess it shouldn't.

Now, this will make the script fail after 20 tries, this is where I use restart: always to simply try again, and again.

You can of course do some clever application code instead of going to bash for help, but personally, I don't always use node, so I love having a generic solution.