[Bug]: Podman - Reaper creation fails

[fpozzobon]

Testcontainers version

0.14.0

Using the latest Testcontainers version?

Yes

Host OS

MacOs

Host Arch

amd64

Go Version

1.18

Docker version

podman 4.0.1

Docker info

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.18-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1611247616
  memTotal: 2061381632
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 12m 43.17s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 42
    paused: 0
    running: 0
    stopped: 42
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 40
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1645808711
  BuiltTime: Fri Feb 25 19:05:11 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/amd64
  Version: 4.0.1

What happened?

When Skipping reaper, Podman works correctly, but when trying to create reaper getting error: Error response from daemon: container create: statfs /var/run/docker.sock: permission denied: creating reaper failed: failed to create container

Relevant log output

Error response from daemon: container create: statfs /var/run/docker.sock: permission denied: creating reaper failed: failed to create container

Additional Information

It seems that something it missing on configuration to be able to create reaper using podman as part of this PR

[fpozzobon]

@baez90 could it be due to https://github.com/testcontainers/testcontainers-go/blob/6ba6e7a0e4b0046507c28e24946d595a65a96dbf/reaper.go#L64 in which NetworkMode Bridge is hard coded?

[mdelapenya]

I think that could be the reason for the reaper not being able to discover the containers in Podman. Would you be interested in contributing a fix? @baez90 please let us know what you think about @fpozzobon's discovery.

[prskr]

@fpozzobon I'd be surprised if the network mode would be a problem regarding recognizing the correct Docker socket for the reaper.

Can you probably provide some more environment information regarding how you configure the Docker socket path in your setup?

It's a bit difficult for me to reproduce as I don't have a Mac 😅

[kiview]

I don't understand the original issue description. It is using Podman on MacOS, but docker info suggests Fedora as host. What is the actual setup?

Also Podman is running on /run/user/1000/podman/podman.sock, not /var/run/docker.sock.

[fpozzobon]

@kiview thanks for your feedback, it is indeed confusing as Podman is actually running via Qemu on Macos (here a description of the architecture). That might be why it is a bit more complex to achieve compared to directly on linux distribution. So takeaway is that the issue is only on MacOS, reaper works fine with Linux?

[prskr]

I think I figured out (by chance) what might be a/the problem and now I also understand what @fpozzobon meant with the fixed bridge network :smile: although I'm really surprised that the current test suite doesn't cover that :smile:

I'm not 100% sure if this is the only issue.

@fpozzobon did you see something like this:

Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container

when you tried running your tests?

I originally was confused by the

/var/run/docker.sock: permission denied: creating reaper failed: failed to create container

message. But that might also indicate that your environment is not 100% configured correctly :sweat_smile:

[yanghua-ola]

I think I figured out (by chance) what might be a/the problem and now I also understand what @fpozzobon meant with the fixed bridge network :smile: although I'm really surprised that the current test suite doesn't cover that :smile:

I'm not 100% sure if this is the only issue.

@fpozzobon did you see something like this:

Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container

when you tried running your tests?

I originally was confused by the

/var/run/docker.sock: permission denied: creating reaper failed: failed to create container

message. But that might also indicate that your environment is not 100% configured correctly :sweat_smile:

I had the same issue with OP; and I recall seeing the error messages you quoted. Did you find a fix?

[yanghua-ola]

Mac OS 12.6, got testcontainers-go and Ryuk working with Podman. Below are the crucial steps:

• podman machine set --rootful before starting
• run reaper in privileged mode

  cat > ~/.testcontainers.properties 
  ryuk.container.privileged = true
  

[mdelapenya]

@yanghua-ola I think this is a great opportunity to improve our podman docs: https://golang.testcontainers.org/system_requirements/using_podman/

Would you willing to contribute it?

[yanghua-ola]

@yanghua-ola I think this is a great opportunity to improve our podman docs: https://golang.testcontainers.org/system_requirements/using_podman/

Would you willing to contribute it?

Would be glad to, will raise a PR later

[fpozzobon]

fyi I found a way around to fix it for me by changing environment variables as follow:

export TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true; # needed to run Reaper (alternative disable it TESTCONTAINERS_RYUK_DISABLED=true)
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock; # needed to apply the bind with statfs

and (sometimes?) make Podman rootful: podman machine set --rootful;

[kstiehl]

fyi I found a way around to fix it for me by changing environment variables as follow:

export TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true; # needed to run Reaper (alternative disable it TESTCONTAINERS_RYUK_DISABLED=true)
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock; # needed to apply the bind with statfs

and (sometimes?) make Podman rootful: podman machine set --rootful;

Still not sure why it works but the DOCKER_SOCKET_OVERRIDE did the trick.

[godart]

Mac OS 12.6, got testcontainers-go and Ryuk working with Podman. Below are the crucial steps:

* `podman machine set --rootful` before starting

* run reaper in privileged mode
  ```
  cat > ~/.testcontainers.properties 
  ryuk.container.privileged = true
  ```

One of these options should be sufficient. You either run all of Podman as root. (Which IMO is defeating the purpose of running Podman at all)

Or you can run the Ryuk container in privileged mode only, via the given setting. I have tried this case and it works perfectly. Would it be possible to make ryuk.container.priviledged default to true? As far as I understand the whole feature of a reaper container does only work, if this special container has full access to the container runtime (Docker/Podman/...).