[Bug]: Substituted Kafka image cannot run in KRaft mode due to version check

[pw-lehre]

Module

Kafka

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host Arch

x64_86

Docker version

$ docker version
Client:
 Version:           24.0.6-rd
 API version:       1.42 (downgraded from 1.43)
 Go version:        go1.20.7
 Git commit:        da4c87c
 Built:             Wed Sep  6 16:41:58 2023
 OS/Arch:           windows/amd64
 Context:           default

Server:
 Engine:
  Version:          23.0.6
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       9dbdbd4b6d7681bd18c897a6ba0376073c2a72ff
  Built:            Thu Aug  3 05:23:23 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.2
  GitCommit:        0cae528dd6cb557f7201036e9f43420650207b58
 runc:
  Version:          1.1.7
  GitCommit:        860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 docker-init:
  Version:          0.19.0
  GitCommit:

What happened?

Testcontainers does not consider substituted Kafka images.

When substituting a Kafka image and then enabling KRaft, the verifyMinKafkaVersion() method does not consider that the image could be substituted (and thus have a different version).

https://github.com/testcontainers/testcontainers-java/blob/9cfa1662d9e9fc3dae619a5250f8c8bb78d67668/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L120-L122

This check doesn't make sense for substituted images - I know that I need a KRaft-compatible Kafka image to run in KRaft mode.

Relevant log output

No response

Additional Information

No response

[yeikel]

I am also seeing this issue

This works:

    static final DockerImageName KAFKA_DOCKER_IMAGE = DockerImageName.parse("confluentinc/cp-kafka:7.4.0@sha256:187dac6627e7906c350f5f8c982f80ce735ff1a0e571a20de6000a309a12ce63")
            .asCompatibleSubstituteFor("confluentinc/cp-kafka:7.4.0");

But this:

    static final DockerImageName KAFKA_DOCKER_IMAGE = DockerImageName.parse("confluentinc/cp-kafka:7.7.0@sha256:1fa3cdb3ef77b75bc56e7040bbdfa2bd6950b6ee6363381b2493f7a83e07a0b3")
            .asCompatibleSubstituteFor("confluentinc/cp-kafka:7.7.0");

Fails with the error :

java.lang.IllegalArgumentException: Provided Confluent Platform's version sha256:1fa3cdb3ef77b75bc56e7040bbdfa2bd6950b6ee6363381b2493f7a83e07a0b3 is not supported in Kraft mode (must be 7.0.0 or above)

[eddumelendez]

I recommend to update to version 1.20.2 and use the new ConfluentKafkaContainer for confluentinc/cp-kafka images or org.testcontainers.kafka.KafkaContainer

[kiview]

@pw-lehre @yeikel Can you please provide feedback if using the new container implementations solve the issue for you? I'll close the issue for now, but we can consider reopening it, if the issues persist.

[yeikel]

@pw-lehre @yeikel Can you please provide feedback if using the new container implementations solve the issue for you? I'll close the issue for now, but we can consider reopening it, if the issues persist.

Hi @kiview I gave it a try, and the issue reported here seems fixed.

At the same time, it is not a full replacement for me. It seems that the new container does not meet my requirement. Namely, the option to start it using SSL overwriting the listeners

That can be a new issue I suppose

[eddumelendez]

Hi, can you share a project that was working and now using the replacement fails? The configuration is the same but the only thing that changes is the port, previous implementation used 9093 and now it uses 9092.