testcontainers-dotnet icon indicating copy to clipboard operation
testcontainers-dotnet copied to clipboard

[Bug]: Cannot connect a Schema Registry running from another container due to missing kafka broker alias

Open leonasa opened this issue 2 years ago • 0 comments

Testcontainers version

2.3.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x64

.NET version

6.0.x

Docker version

Client: Docker Engine - Community
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:02:28 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:19 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.13.0)
  dev: Docker Dev Environments (Docker Inc., v0.0.5)
  extension: Manages Docker extensions (Docker Inc., v0.2.16)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.22.0)

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 33
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.79.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.48GiB
 Name: docker-desktop
 ID: Q5DY:AEVR:PB2K:53ON:CME5:WM2R:GH3I:R5CR:RQQQ:MXMA:FYFN:C7W3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

What happened?

I was trying to connect a kafka broker to schema registry running on Testcontainers using the following:

var kafka = new TestcontainersBuilder<KafkaTestcontainer>()
    .WithNetwork(network)
    .WithNetworkAliases(kafkaContainerNetworkAliases)
    .WithKafka(new KafkaTestcontainerConfiguration())
    .Build();

and

var schemaRegistryContainer = new TestcontainersBuilder<TestcontainersContainer>()
    .WithImage("confluentinc/cp-schema-registry:latest")
    .WithPortBinding(8081, true)
    .WithNetwork(network)
    .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(8081))
    .WithEnvironment("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", kafka.BootstrapServers)
    .WithEnvironment("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
    .WithEnvironment("SCHEMA_REGISTRY_HOST_LISTENERS", "http://0.0.0.0:8081")
    .Build();

Relevant log output

No response

Additional information

The runnning kafka broker need the network alias to allow other containers to connect internally. See: Kafka Listerners explained

Previously generating the alias would fix the problem:

var kafkaContainerNetworkAliases = Guid.NewGuid().ToString();

var kafkaTestcontainerConfiguration = new KafkaTestcontainerConfigurationNew(kafkaContainerNetworkAliases);
        
var kafka = new TestcontainersBuilder<KafkaTestcontainer>()
    .WithNetwork(network)
    .WithPortBinding(9092)
    .WithNetworkAliases(kafkaContainerNetworkAliases)
    .WithKafka(kafkaTestcontainerConfiguration)
    .Build();

var schemaRegistryContainer = new TestcontainersBuilder<TestcontainersContainer>()
    .WithImage("confluentinc/cp-schema-registry:latest")
    .WithPortBinding(8081, true)
    .WithNetwork(network)
    .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(8081))
    .WithEnvironment("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", kafkaTestcontainerConfiguration.KafkaBrokerInternalUrl)
    .WithEnvironment("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
    .WithEnvironment("SCHEMA_REGISTRY_HOST_LISTENERS", "http://0.0.0.0:8081")
    .Build();

leonasa avatar Jan 16 '23 16:01 leonasa