testcontainers
Bug: RYUK container startup failure when running Docker in rootless mode
Describe the bug
When Docker is run in rootless mode, the ryuk fails to start as the docker socket mounted as a volume has the wrong permissions (nobody:nobody). This can be fixed be mounting the correct socket at /run/user/$(id -u)/docker.sock.
The overwrite can be done by setting the environment variable TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE to the above mentioned socket.
Ideally, this would be detected and configured automatically (ie. using Server.Security Options.rootless and Endpoints.docker.Host of the active context or the DOCKER_HOST environment variable).
If this is not an option, please consider documenting this case more in detail.
To Reproduce
Configure Docker to run in rootless mode (see https://docs.docker.com/engine/security/rootless/) and run the following snippet:
>>> from testcontainers.postgres import PostgresContainer
>>> with PostgresContainer() as container:
... assert True
Runtime environment
Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.
# Get the operating system information (on a unix os).
$ uname -a
Linux FE-C-012RG 6.5.0-1019-oem #20-Ubuntu SMP PREEMPT_DYNAMIC Mon Mar 18 17:38:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# Get the python version.
$ python --version
Python 3.12.2
# Get the docker version and other docker information.
$ docker info
Client: Docker Engine - Community
Version: 26.0.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.13.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.26.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 19
Server Version: 26.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e377cd56a71523140ca6ae87e30244719194a521
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: builtin
rootless
cgroupns
Kernel Version: 6.5.0-1019-oem
Operating System: Ubuntu 22.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.32GiB
Name: FE-C-012RG
ID: 547c075a-39fa-4b4a-950a-b18069861839
Docker Root Dir: /home/CFL5FE/.local/share/docker
Debug Mode: false
Username: keneanung
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
# Get all python packages.
$ pip freeze
alembic==1.13.1
annotated-types==0.6.0
anyio==4.3.0
astroid==3.1.0
asyncpg==0.29.0
-e <my local repo>
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
coverage==7.4.4
cryptography==42.0.5
dill==0.3.8
docker==7.0.0
ecdsa==0.19.0
fastapi==0.110.1
fastapi-azure-auth==4.3.1
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
hvac==2.1.0
idna==3.6
importlib_metadata==7.1.0
iniconfig==2.0.0
isort==5.13.2
Mako==1.3.2
MarkupSafe==2.1.5
mccabe==0.7.0
packaging==24.0
platformdirs==4.2.0
pluggy==1.4.0
pyasn1==0.6.0
pycparser==2.22
pydantic==2.6.4
pydantic_core==2.16.3
pylint==3.1.0
pytest==8.1.1
pytest-asyncio==0.23.6
pytest-cov==5.0.0
python-jose==3.3.0
PyYAML==6.0.1
requests==2.31.0
rsa==4.9
six==1.16.0
sniffio==1.3.1
SQLAlchemy==2.0.29
starlette==0.37.2
syrupy==4.6.1
testcontainers==4.3.3
tomli==2.0.1
tomlkit==0.12.4
typing_extensions==4.11.0
urllib3==2.2.1
uvicorn==0.29.0
vault-env-gen==0.2.0
wrapt==1.16.0
yapf==0.40.2
zipp==3.18.1
can you confirm this approach:
mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat > detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection
def is_rootless(client: DockerClient):
info = client.info()
sec_opts = info.get('SecurityOptions') or tuple()
return any('rootless' in s for s in sec_opts)
if __name__ == "__main__":
print(is_rootless(from_env()))
EOF
python detect_rootless.py
and then i guess we will need to tweak the Reaper class a bit in core
can you confirm this approach:
mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate pip install docker cat > detect_rootless.py <<EOF from docker import from_env from docker.client import DockerClient from docker.models.containers import Container, ContainerCollection def is_rootless(client: DockerClient): info = client.info() sec_opts = info.get('SecurityOptions') or tuple() return any('rootless' in s for s in sec_opts) if __name__ == "__main__": print(is_rootless(from_env())) EOF python detect_rootless.py
This returns the following:
$ python detect_rootless.py
True
I also use rootless Docker and I can confirm this issue.
I can also confirm that setting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock fixes the issue. Alternatively, I have tried disabling Ryuk through TESTCONTAINERS_RYUK_DISABLED=true, which makes the tests run flawlessly as well.
Would it be possible to fix this so that testcontainers just works with rootless docker? @alexanderankin
TESTCONTAINERS_RYUK_DISABLED=true pytest works, but it's a pain to use, and we can't even just os.environ['TESTCONTAINERS_RYUK_DISABLED'] = 'true' inside the python test file
Nowadays any dev slightly conscious should be using rootless docker (because it is safer, and because it reduce friction with file ownerships when mounting volumes inside an image that use the root user. So not only it is safer, but it is also easier to use...). It's a bit the future of containers, but it's already there and working. So testcontainers should be available for rootless docker without the need for complex setup
you can import the config module and set it on the dataclass there
https://github.com/testcontainers/testcontainers-python/blob/85d6078f9bcc99050c0173e459208402aa4f5026/core/testcontainers/core/config.py#L51
- I'm not in a position where I'm forced to use rootless docker so I dont have easy access to an environment where I could easily test. if there is a guide or something that I can set up, this would be good prereq for contributions imo
Ah, we have confirmed that the logic i provided above works. Then all that remains is plugging it in I suppose. I'll accept PR that fixes the issue. yes.
I am not forced to use docker rootless neither, it's just that it's better than the root option, especially when you use containers a lot for development, try it you'll see :) Rootless docker can be easily enabled following these docs: https://docs.docker.com/engine/security/rootless/ usually 3 commands:
dockerd-rootless-setuptool.sh install
systemctl --user enable docker
loginctl enable-linger $UID
Unfortunately I just realized that ryuk is needed to stop the containers after running the tests. So disabling it is not a sustainable solution
But TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock works fine
An easy solution would be to just change the default RYUK_DOCKER_SOCKET depending on if rootless is detected or not here: https://github.com/testcontainers/testcontainers-python/blob/main/core/testcontainers/core/config.py#L15
I might look into this if I find sometimes and will send a PR
import docker
client = docker.from_env()
print(client.api.get_adapter(client.api.base_url).socket_path)
Could be used to detect the correct socket_path.
If socket_path is not defined, just use the default.
