Use apache/kafka:4.0.0 docker image in tests

[StFS]

Use the latest version of the apache/kafka Docker image (4.0.0) instead of using the confluentinc/cp-kafka image.

The confluent/cp-kafka image does not follow the Kafka version numbers, instead it followed the confluent platform versioning which made it difficult to identify which version of Kafka was being run in the integration tests.

[sophokles73]

I have read through the Kafka 4.0.0 release announcement and to me it looks like we can safely migrate to Kafka 4.0.0 on the broker side. I am not sure if we'll need to adapt our custom KafkaConsumer when updating the client library.

@calohmn any thoughts?

[calohmn]

As for a Kafka client library update, first the vertx-kafka client needs to be compatible with Kafka 4.0. Looking at the consumer changes in 4.0 concerning the removal of previously marked-deprecated methods, there are still cases of such methods being used in vertx-kafka (e.g. consumer.committed(topicPartition) in KafkaReadStreamImpl.java).

Concerning the usage of apache/kafka:4.0.0 in the tests: Up until now, I think we used image versions corresponding to the Kafka version used in Hono. The currently used 3.7.0 version would mean docker.io/apache/kafka:3.7.2 or docker.io/confluentinc/cp-kafka:7.7.4 (based on the confluent version mapping here). (I think in the past we used the confluent images because they were released more frequently and we looked for the inclusion of specific fixes.) Is there a specific reason for already going to the image version 4.0 here?

[StFS]

The reason for me personally was to just verify whether Hono would handle running on 4.x.

But in general, I think it would be preferable if Hono made an attempt to stay up-to-date with the latest version of Kafka because generally, Kafka clients are backwards compatible, meaning that if a company is running an older version of a Kafka cluster (3.7, 3.9 for example), newer kafka client versions (such as those supporting 4.x) are likely to be compatible with those clusters. However, if a company is running the latest version of a Kafka cluster (4.0), older Kafka client versions (3.x) may not be compatible.

So, since Kafka cluster setups are quite expensive (as is the case for MSK), companies are likely to try to share the cluster setup between their own in-house components and components such as Hono. If, for some reason, they feel the need to upgrade to Kafka 4.0, Hono would hold back that upgrade whereas if Hono supported Kafka 4.x, it would work on a 4.x setup but in all likelyhood, it would also work on a 3.x setup.

[sophokles73]

Up until now, I think we used image versions corresponding to the Kafka version used in Hono. The currently used 3.7.0 version would mean docker.io/apache/kafka:3.7.2 or docker.io/confluentinc/cp-kafka:7.7.4 (based on the confluent version mapping here).

Based on that, we should at least update to the 7.9.x version of the confluent Kafka image, in order to match the Kafka client 3.9.1 that we are using at the moment.