[Bug]: Ryuk does not work with podman

[dfioravanti]

Testcontainers version

0.28

Using the latest Testcontainers version?

Yes

Host OS

MacOs

Host arch

ARM

Go version

1.22

Docker version

Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.41 (downgraded from 1.44)
 Go version:        go1.21.6
 Git commit:        4debf411d1
 Built:             Tue Feb  6 20:42:40 2024
 OS/Arch:           darwin/arm64
 Context:           default

Server: linux/arm64/fedora-39
 Podman Engine:
  Version:          4.9.0
  APIVersion:       4.9.0
  Arch:             arm64
  BuildTime:        2024-01-24T11:07:09+01:00
  Experimental:     false
  GitCommit:
  GoVersion:        go1.21.6
  KernelVersion:    6.7.4-200.fc39.aarch64
  MinAPIVersion:    4.0.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.8, commit:
  Package:          conmon-2.1.8-2.fc39.aarch64
 OCI Runtime (crun):
  Version:          crun version 1.14
commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  Package:          crun-1.14-1.fc39.aarch64
 Engine:
  Version:          4.9.0
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:
  Built:            Wed Jan 24 11:07:09 2024
  OS/Arch:          linux/arm64
  Experimental:     false

Docker info

Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 4.9.0
 Storage Driver: overlay
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: false
  Using metacopy: true
  Supports shifting: true
  Supports volatile: true
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge macvlan ipvlan
  Log: k8s-file none passthrough journald
 Swarm: inactive
 Runtimes: kata krun runj youki crun crun-wasm ocijail runc runsc
 Default Runtime: crun
 Init Binary:
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 6.7.4-200.fc39.aarch64
 Operating System: fedora
 OSType: linux
 Architecture: arm64
 CPUs: 4
 Total Memory: 1.904GiB
 Name: localhost.localdomain
 ID: c9c1ca7e-0a95-4d0d-b573-b82806d74d03
 Docker Root Dir: /var/lib/containers/storage
 Debug Mode: false
 Experimental: true
 Live Restore Enabled: false
 Product License: Apache-2.0

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

What happened?

I wanted to switch from docker-desktop to podman as docker provider on MacOs. Everything works but only if I disable ryuk. Otherwise testcontainer crashes at boot up with

getting request provider: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container

Any idea how to fix this? I have tried everything that I can google, podman is run as rootful and the socket is correctly configured. Testcontainers with ryuk disabled works as expected but it would be nice if it would just work without requiring disabling it.

Relevant log output

No response

Additional information

I am using this project to trigger the issue in case you want to reproduce: https://github.com/dfioravanti/go-rest. Just run go test ./... in the root.

[godart]

Running into the same issue. The test I used to recreate the issue looks like:

import (
	"context"
	"testing"

	"github.com/testcontainers/testcontainers-go"
)

func TestPodman(t *testing.T) {
	ctx := context.Background()
	_, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{}, // never evaluated ...
	})
	if err != nil {
		t.Fatalf("could not start container: %v", err)
	}
}

I'd expect it to fail with something hinting towards the image name being empty.

The output is

=== RUN   TestPodman
2024/04/17 12:04:54 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 5.0.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 1952 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 748e590ad3a32566931082a26236b0bed5703256a2a60b218eb5f4bfd72a9b76
  Test ProcessID: 6986206b-fbed-4e04-8f4e-d2ba49e32112
2024/04/17 12:04:54 🐳 Creating container for image testcontainers/ryuk:0.7.0
    model_test.go:36: could not start container: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container
--- FAIL: TestPodman (0.09s)
=== RUN   TestPodman
2024/04/17 12:04:54 🐳 Creating container for image testcontainers/ryuk:0.7.0
    podman_testcontainers_test.go:16: could not start container: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container
--- FAIL: TestPodman (0.01s)


FAIL

using Podman 5.0.1 here:

~> podman version
Client:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.22.1
Git Commit:   946d055df324e4ed6c1e806b561af4740db4fea9
Built:        Mon Apr  1 13:28:31 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.21.8
Built:        Mon Apr  1 01:00:00 2024
OS/Arch:      linux/arm64

[godart]

I debugged a bit:

The current auto-detection of Podman looks for "podman.sock" within the socket name. This works with Podman on Linux, but when using podman-mac-helper for MacOS, the socket name (on the host) is /var/run/docker.sock. (https://github.com/testcontainers/testcontainers-go/blob/main/provider.go#L106)

Setting ProviderType in the test code like this

	_, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{}, // never evaluated ...
		ProviderType:     testcontainers.ProviderPodman,
	})

fixes the network configuration issue. Also the PR https://github.com/testcontainers/testcontainers-go/pull/1990 addresses this with a global configuration option. However this has been stale for a bit.

Maybe the /info or /version calls could be scraped to detect the server is actually Podman instead.

[MadsRC]

I ran into this issue on a setup similar to the original poster.

@godart, I tried your fix, but I kept running into this issue:

2024/05/04 21:42:45 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 5.0.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 3622 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: f9ea1ec477217d1ab405cd38e21e3a471f5360d6b852712ecd8c286ba9a5f9f2
  Test ProcessID: 9043d990-9cb6-40d3-adf7-ba8d0381a833
2024/05/04 21:42:45 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/05/04 21:42:45 ✅ Container created: 2b3afd5d2e60
2024/05/04 21:42:45 🐳 Starting container: 2b3afd5d2e60
2024/05/04 21:42:45 ✅ Container started: 2b3afd5d2e60
2024/05/04 21:42:45 🚧 Waiting for container id 2b3afd5d2e60 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/05/04 21:42:45 container logs (unexpected container status "stopped"):
2024/05/04 19:42:45 Pinging Docker...
panic: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

I even tried setting my DOCKER_HOST to point to my podman socket with DOCKER_HOST=unix:///Users/user/.local/share/containers/podman/machine/podman.sock which got rid of the permissions issue but still yielded an error:

2024/05/04 21:56:15 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 5.0.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 3622 MB
  Resolved Docker Host: unix:///Users/user/.local/share/containers/podman/machine/podman.sock
  Resolved Docker Socket Path: /Users/user/.local/share/containers/podman/machine/podman.sock
  Test SessionID: 64a4a58cee7888739aebd93541d7a4b0b0200d84a9b10f396a5c4148984f1601
  Test ProcessID: 33aa8c36-5cdb-4b96-8725-b9539ef9c0d3
2024/05/04 21:56:15 🐳 Creating container for image testcontainers/ryuk:0.7.0
--- FAIL: TestWithRedis (0.07s)
    startup_test.go:23: Could not start redis: Error response from daemon: container create: statfs /Users/user/.local/share/containers/podman/machine/podman.sock: operation not supported: creating reaper failed: failed to create container

Did you not run into this issue?

[mohsanabbas]

We are having similar issues with colima and rancher-desktop

    2024/05/09 11:11:09 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 24.0.9
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 1895 MB
  Resolved Docker Host: unix:///Users/mohsan.abbas/.colima/default/docker.sock
  Resolved Docker Socket Path: /Users/mohsan.abbas/.colima/default/docker.sock
  Test SessionID: c1729139638928699b678e3fdc7dfff4e56482cc2310ee2358a09f68931641e0
  Test ProcessID: e4f04ff7-5395-48a9-be14-51c54e9af8df
2024/05/09 11:11:09 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/05/09 11:11:09 ✅ Container created: 114d4680aa73
2024/05/09 11:11:09 🐳 Starting container: 114d4680aa73
2024/05/09 11:11:09 ✅ Container started: 114d4680aa73
2024/05/09 11:11:09 🚧 Waiting for container id 114d4680aa73 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/05/09 11:11:09 failed accessing container logs: Error response from daemon: No such container: 114d4680aa73b82030172ccfe23cb81d4a7364beb7217e6fb2472cd716b0d055
--- FAIL: TestKafkaIntegration (0.57s)
    kgo_test.go:182: Failed to start Kafka container: Error response from daemon: No such container: 114d4680aa73b82030172ccfe23cb81d4a7364beb7217e6fb2472cd716b0d055: could not start container: creating reaper failed: failed to create container
FAIL