testcontainers-go
testcontainers-go copied to clipboard
testcontainers
[Bug]: Ryuk container created, then it is removed before it starts
Testcontainers version
0.31.0
Using the latest Testcontainers version?
Yes
Host OS
Mac OS
Host arch
ARM
Go version
1.22
Docker version
Client: Docker Engine - Community
Version: 26.1.4
API version: 1.45
Go version: go1.22.4
Git commit: 5650f9b102
Built: Wed Jun 5 10:47:13 2024
OS/Arch: darwin/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.1
API version: 1.45 (minimum version 1.24)
Go version: go1.21.9
Git commit: ac2de55
Built: Tue Apr 30 11:48:47 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Docker info
Client: Docker Engine - Community
Version: 26.1.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.15.0
Path: /Users/river/.docker/cli-plugins/docker-buildx
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 3
Server Version: 26.1.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e377cd56a71523140ca6ae87e30244719194a521
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-31-generic
Operating System: Ubuntu 24.04 LTS
OSType: linux
Architecture: aarch64
CPUs: 2
Total Memory: 1.91GiB
Name: colima
ID: 5a6217f8-7cca-498b-91f6-d4070390a0e4
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
What happened?
Any attempt to run a test using test containers appears to create the Ryuk Container, return an ID. Then it is removed while waiting for the container to start.
Relevant log output
024/06/18 16:26:45 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/06/18 16:26:45 ✅ Container created: 4879564c4f90
2024/06/18 16:26:45 🐳 Starting container: 4879564c4f90
2024/06/18 16:26:45 ✅ Container started: 4879564c4f90
2024/06/18 16:26:45 ⏳ Waiting for container id 4879564c4f90 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/06/18 16:26:45 failed accessing container logs: Error response from daemon: No such container: 4879564c4f9093fffa6b43a4a75d48e63a71672bc391fd3ce69951bb13152643
Additional information
I attempted to run test tests with main and observed the same failures so I don't believe this fixes it. https://github.com/testcontainers/testcontainers-go/issues/2477
Experiencing the same issue. macOS 14.5 Sonoma, Rancher Desktop 1.14.2. Testcontainers 36573d0adcf650207fb9b8c61072518a84a09674 (a few commits after 0.31.0).
Client: Docker Engine - Community
Version: 26.1.4
Context: rancher-desktop
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.14.1
Path: /Users/mike/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.27.1
Path: /Users/mike/.docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 25.0.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4e1fe7492b9df85914c389d1f15a3ceedbb280ac
runc version: 51d5e94601ceffbbd85688df1c928ecccbfa4685
init version:
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.6.30-0-virt
Operating System: Alpine Linux v3.19
OSType: linux
Architecture: aarch64
CPUs: 9
Total Memory: 15.6GiB
Name: lima-rancher-desktop
ID: d04c0365-7614-447f-960d-353ba1523b4f
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
It worked yesterday, not sure what I did to break it.
Disabling Ryuk isn't really a solution, we'd like to continue to use it.
Hi 👋 are you by the way using the compose module? If so please take a look at https://github.com/testcontainers/testcontainers-go/issues/2563#issuecomment-2154596449
If not, I'd like to see the ryuk logs if possible, so we can determine why it's being removed. Is it a timeout issue? You can check the ryuk configuration here: https://golang.testcontainers.org/features/configuration/#customizing-ryuk-the-resource-reaper
Hi @mdelapenya we're not using compose.
Ryuk doesn't appear to log anything in this case. Upon further inspection when I run go test from the command line it does work, but running it from VS code's test runner, or Goland's test runner it fails.
Mmm that's even weirder. Can you verify that the IDE sets the proper go settings for testing, such as "go.testTimeout" (I configured it to 600s)? I wonder why the ryuk container dies in VSCode but not in your terminal.
Can you provide a repro repository?
Mmm that's even weirder. Can you verify that the IDE sets the proper go settings for testing, such as "go.testTimeout" (I configured it to 600s)? I wonder why the ryuk container dies in VSCode but not in your terminal.
The test timeouts look good, it fails almost instantly, it's not taking a long time.
Can you provide a repro repository?
I have this issue for any project using testcontainers
This should be fixed by https://github.com/testcontainers/moby-ryuk/pull/121 but we need a release and then testcontainers-go updated to use the new release.
Try cloning the moby-ryuk repo and running the following in it to replace the image that testcontainers-go uses to see if it does fix:
docker build -f linux/Dockerfile -t testcontainers/ryuk:0.7.0 .
Thanks for looking into this, I've updated to use main and I sometimes get the same error, and sometimes another error, I think it's still related. @mdelapenya are you able to re-open this please, or should I file another issue?
Error: Received unexpected error:
unexpected container status "removing": could not start container: creating reaper failed: failed to create container
Thanks for testing.
I have a bug bash branch that I'm working on improving the stability of tests which includes a reworked reaper spawner and while this may not fix the issue, it also includes improvements to the errors so might help point to where the underlying issue is in your case.
If you could test on the branch from https://github.com/testcontainers/testcontainers-go/pull/2664 and let me know what you get that would be appreciated.
If you could also post a test which reproduces the issue that would help too.
I fixed this issue modifying the ~/.zshrc file with this configuration
export DOCKER_HOST="unix:///Users/${HOME}/.colima/default/docker.sock"
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE="/var/run/docker.sock"
I fixed this issue modifying the ~/.zshrc file with this configuration
export DOCKER_HOST="unix:///Users/${HOME}/.colima/default/docker.sock" export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE="/var/run/docker.sock"
This is a weird solution, how does this work? It worked for me as well (as disabling RYUK), but afaiu I don't have any docker running on /var/run/docker.sock as I am using Rancher Desktop, which socket is also in a different path.
@NanoBjorn @joseroblesv that's not even the weirdest part. It's that your ${HOME} is probably an absolute path in /Users/. Thus, this:
export DOCKER_HOST="unix:///Users/${HOME}/.colima/default/docker.sock"
will lead to something like unix:///Users//Users/<your_username>/.colima/default/docker.sock, right? Can you check what's going on there with your $HOME variable? Also, maybe that works because it's using a fallback or something?
@timofurrer I used absolute path without env variables when I was testing it. Might be that it was a typo from @joseroblesv as well, because otherwise testcontainers would not be able to run containers.