Rolling upgrade cp-kafka 4.0.0-3 to 4.1.1-1

[mariuscoto]

In a 3 broker Kafka cluster when trying a rolling upgrade from 4.0.0-3 to 4.1.1-1, after rebooting the first broker I see the following message in the logs:

[2018-05-23 11:54:36,629] WARN [ReplicaFetcher replicaId=1003, leaderId=1002, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=1003, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=843861464, epoch=13061)) (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1002 was disconnected before the response was read
	at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
	at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:96)
	at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:220)
	at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:43)
	at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:146)
	at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

while the rest of the brokers report the following exception:

[2018-05-23 10:00:35,002] ERROR Exception while processing request from 127.0.0.1:39092-127.0.0.1:38038-59 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: FETCH, apiVersion: 7, connectionId: 127.0.0.1:39092-127.0.0.1:38038-59, listenerName: ListenerName(PLAINTEXT), principal: User:ANONYMOUS
Caused by: java.lang.IllegalArgumentException: Invalid version for API key FETCH: 7
	at org.apache.kafka.common.protocol.ApiKeys.schemaFor(ApiKeys.java:297)
	at org.apache.kafka.common.protocol.ApiKeys.requestSchema(ApiKeys.java:267)
	at org.apache.kafka.common.protocol.ApiKeys.parseRequest(ApiKeys.java:275)
	at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:63)
	at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:85)
	at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:551)
	at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:545)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at kafka.network.Processor.processCompletedReceives(SocketServer.scala:545)
	at kafka.network.Processor.run(SocketServer.scala:453)
	at java.lang.Thread.run(Thread.java:745)

When rebooting the last broker in the cluster to the new version, there is a short downtime period, until the broker gets up again, during which the consumer and producers cannot access Kafka.

What can I do in order to have a smooth upgrade process ?

[vitaliis]

Hi @mariuscoto ,

we had the same issue, but with pure kafka setup (also 3 brokers/servers). After some investigation we solved it by adding an additional configuration setting to the broker's settings. In our case we upgraded from 1.0.0 to 1.1.0 and added the following to each broker's config:

inter.broker.protocol.version=1.0

Just to be clear, you don't have to modify existing brokers, but this setting has to be rolled out with new brokers (of version 1.1.0).

More information can be found here: http://kafka.apache.org/11/documentation.html#upgrade_1_1_0

[mariuscoto]

That worked. Thank you!

[ethicalmohit]

@mariuscoto This setting is not there in the kafka.properties of the broker. Have you explicitly mentioned this via environmental variable?

[mariuscoto]

@ethicalmohit yes, I've set the KAFKA_INTER_BROKER_PROTOCOL_VERSION environment variable