testcontainers-go icon indicating copy to clipboard operation
testcontainers-go copied to clipboard

fix(modules.kafka): Use broker container IP instead of host IP for advertised broker listener

Open khartld opened this issue 1 year ago • 2 comments

What does this PR do?

This PR changes the advertised broker ip from being the host ip to container ip. Port 9092 is not exposed to the host, so we can not connect to it via host IP, but need to use the broker container IP. The current setup works with a normal desktop setup, because there the returned host IP is "localhost".

Alternatively, it should also be sufficient to hardcode the advertised broker as "localhost", because there is only 1 broker that needs to be able to connect to itself.

Why is it important?

This makes it possible to use the Kafka module inside another Docker container or other setups where Host() does not return "localhost". This enables the use of this module in CI/CD pipelines that run in Docker.

Currently it leads to errors in the TransactionCoordinator: WARN [TransactionCoordinator id=1] Connection to node 1 (/172.17.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Related issues

  • Closes #1786 This PR tackles the same problem, but using the container name instead of ip breaks the normal Desktop setup, as the system doesn't know the container name as a hostname.

How to test this PR

After starting a Kafka container using the Kafka module, the KAFKA_ADVERTISED_LISTENERS should contain the kafka container ip.

E.g. the /usr/sbin/testcontainers_start.sh script should contain the following value:

export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:51702,BROKER://172.17.0.6:9092

khartld avatar Dec 08 '23 14:12 khartld

Deploy Preview for testcontainers-go ready!

Name Link
Latest commit f4991bab4bbabc79f4d99be25ed155ac14c93a9a
Latest deploy log https://app.netlify.com/sites/testcontainers-go/deploys/663b16229b5898000892fbcc
Deploy Preview https://deploy-preview-1989--testcontainers-go.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] avatar Dec 08 '23 14:12 netlify[bot]

@mdelapenya @eddumelendez what do you think about merging this? :)

khartld avatar Mar 15 '24 14:03 khartld

@eddumelendez, as our kafka expert, what are your thoughts on this?

mdelapenya avatar Apr 05 '24 22:04 mdelapenya

@khartld I went ahead and resolved conflicts and rebased your PR with main, as we already merged #2534

Please let us know what you think

Thanks!

mdelapenya avatar May 07 '24 15:05 mdelapenya

@mdelapenya looks good, thanks :)

khartld avatar May 08 '24 12:05 khartld