[Bug]: Some tests are not working on Podman due to Ryuk

[vchandela]

Testcontainers version

0.33.0

Using the latest Testcontainers version?

Yes

Host OS

Mac

Host arch

ARM

Go version

1.22

Docker version

Client:       Podman Engine
Version:      5.2.0
API Version:  5.2.0
Go Version:   go1.22.5
Git Commit:   b22d5c61eef93475413724f49fd6a32980d2c746
Built:        Fri Aug  2 17:37:40 2024
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.6
Built:        Wed Aug 21 05:30:00 2024
OS/Arch:      linux/amd64

Docker info

host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 98.65
    systemPercent: 0.7
    userPercent: 0.65
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2046
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.9.12-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1505406976
  memTotal: 1908293632
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.20240819115418474394.main.6.gc2cd0be.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0-dev
    package: netavark-1.12.1-1.20240819170533312370.main.26.g4358fd3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0-dev
  ociRuntime:
    name: crun
    package: crun-1.16-1.20240813143753154884.main.16.g26c7687.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 1c1550ad8b233275d6ef04d60003b3c59bf42d71
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240726.g57a21d2-1.fc40.x86_64
    version: |
      pasta 0^20240726.g57a21d2-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 39m 19.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 9058627584
  graphRootUsed: 2671116288
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Wed Aug 21 05:30:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

What happened?

• I tried running all the tests on podman. Some don't work as ryuk is enabled.
• After disabling ryuk using: os.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true") the tests work fine.
• Maybe, we can check if the system has podman and disable ryuk for them as podman is quite heavily used by Linux users.
• Will help in onboarding more contributors to the project.

Relevant log output

Error:          Received unexpected error:
                                create container: container create: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed

Additional information

Alternatively, we can simply mention in the docs that everyone has to export TESTCONTAINERS_RYUK_DISABLED=true before running the tests on Podman.

[kiview]

Thanks for the investigations and raising the issue @vchandela.

Just to give some context, we only officially support Docker as the runtime. We don't intend to add Podman-specific code to any of the Testcontainers language implementation. However, it is fine to add overall more resilient behavior and graceful-degredation behavior based on discovered capabilities of the detected container runtime configuration.

Indeed, in tc-java, we fallback to a JVM based Ryuk implementation, if container-based Ryuk is disabled: https://github.com/testcontainers/testcontainers-java/blob/f0dd61404fd570b91e71513317bde6a5a2599dfa/core/src/main/java/org/testcontainers/utility/ResourceReaper.java#L75-L93

So, I would be open to probe the container runtime for Ryuk support (not specific Podman support, since there are also possible Docker installations that don't support Ryuk) and then disable it dynamically in a graceful manner.

[vchandela]

Thanks for the investigations and raising the issue @vchandela.

Just to give some context, we only officially support Docker as the runtime. We don't intend to add Podman-specific code to any of the Testcontainers language implementation. However, it is fine to add overall more resilient behavior and graceful-degredation behavior based on discovered capabilities of the detected container runtime configuration.

Indeed, in tc-java, we fallback to a JVM based Ryuk implementation, if container-based Ryuk is disabled: https://github.com/testcontainers/testcontainers-java/blob/f0dd61404fd570b91e71513317bde6a5a2599dfa/core/src/main/java/org/testcontainers/utility/ResourceReaper.java#L75-L93

So, I would be open to probe the container runtime for Ryuk support (not specific Podman support, since there are also possible Docker installations that don't support Ryuk) and then disable it dynamically in a graceful manner.

@kiview sorry for the confusion. Yeah, we should investigate why Ryuk doesn't work with Podman. Also, I've updated the title of the issue now.

[seanlafferty-ibm]

The default podman network is podman, while in docker it is bridge. By default, TestContainers is trying to use bridge which won't exist if you're using podman.

You can set ProviderType: testcontainers.ProviderPodman on your GenericContainerRequest to get around this.

The next problem you'll likely run into is that the reaper needs --privileged in podman, which you can do with os.Setenv("TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED", "true")

[stevenh]

The provider type will auto detect podman if DOCKER_HOST is set, does that help?

[fxkk]

After a few experiments, I can contribute another perspective that may be helpful for some people. However, it is about the combination Podman/MacOS which, as described above, is not the focus of the testcontainers project. I hope the hint is still appropriate here.

The two steps outlined so far do not seem to be sufficient in this case.

The final problem is that the privileged Ryuk container inside the Podman Machine must also be able to mount the socket using the same path.

You can solve this by using a path as Docker-Host that is also available inside the Podman Machine.

# verify socket path inside podman machine
$ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 40.20240808.2.0

root@localhost:~# ls -al /var/run/podman/podman.sock 
srw-rw----. 1 root root 0 Dec 20 14:32 /var/run/podman/podman.sock
exit

# On the host machine
$ sudo mkdir /var/run/podman
$ sudo ln -s /Users/Your.User/.local/share/containers/podman/machine/podman.sock /var/run/podman/podman.sock

export DOCKER_HOST="unix:///var/run/podman/podman.sock"
export TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true

I hope this is not too off-topic and that it helps someone.