Adding support for custom DOCKER_HOST / Rootless docker

[srini-hv]

We are in process of adopting Rootless Docker for all our self-hosted runners for better security and isolation. While testing our workflows and custom actions, we noticed that Runner is not acknowledging the DOCKER_HOST Environment variable for Docker-in-Docker Actions and Docker Container Jobs, even when it is set on .env file on the runner directory.

In our case, DOCKER_HOST=unix:///run/user/2022/docker.sock

I noticed that docker socket /var/run/docker.sock is hard-coded in the runner code for both these scenarios and I tried to make these dynamic by reading it from Environment variable in this PR. This change would be technically fixing #827

PS: I'm new to C# and if there is any better way to do this, I'm happy to update the PR.

[fhammerl]

Hi @srini-hv 👋 Thanks for the PR. We'll have to evaluate how this might affect users who may have DOCKER_HOST set to something on their environment, but have come to expect the runner to ignore it and use /var/run/docker.sock instead.

[srini-hv]

Hello @fhammerl . You are right, that could be a scenario. What if we use a dedicated environment variable for the socket file? This would be different for people using Rootless docker for sure. Like DOCKER_SOCKET may be? If the user's docker is running anywhere else other than /var/run/docker.sock, they specify the path using this new environment variable. Does that sound acceptable?

[NunoMaga]

+1

[Kurt-von-Laven]

For folks wanting to run Docker in rootless mode on GitHub Actions, I recommend giving ScribeMD/rootless-docker a try. Full disclosure: I am the author of this action.

[jmcrfp]

We are already running Docker in rootless mode. The issue is when trying to run docker in docker(rootless)

[Kurt-von-Laven]

Apologies for my careless reading. We recently began encountering this same issue with rootless Docker (not dind). We were able to work around it by proxying bidirectionally between unix:///var/run/docker.sock and the rootless Docker socket (e.g., unix:///run/user/1001/docker.sock or more generally unix://$XDG_RUNTIME_DIR/docker.sock). Here is the relevant snippet of rootless-docker in case it helps you or others. You may alternatively be able to simply run the rootless-docker action and then perform any additional needed Docker setup afterwards.

[NunoMaga]

Apologies for my careless reading. We recently began encountering this same issue with rootless Docker (not dind). We were able to work around it by proxying bidirectionally between unix:///var/run/docker.sock and the rootless Docker socket (e.g., unix:///run/user/1001/docker.sock or more generally unix://$XDG_RUNTIME_DIR/docker.sock). Here is the relevant snippet of rootless-docker in case it helps you or others. You may alternatively be able to simply run the rootless-docker action and then perform any additional needed Docker setup afterwards.

our issue is that we are unable to mount the rootless docker socket in the docker run command. Are you suggesting we proxy the socket at runner install? and if so, how did you go about doing so? (cannot open the provided relevant snippet)

[Kurt-von-Laven]

What do you mean by runner install? I don't know whether I accurately understand what you are trying to do, but it should be quick to try bidirectionally proxying the rootful and rootless Docker sockets, and I can't think of a specific reason it would fix our problem but not yours.

rootless-docker is a public repository, and I am able to open the link in an incognito window. What happens when you click on the link? Here is the snippet in any case:

    - name: Proxy bidirectionally between rootful and rootless Docker sockets.
      if: steps.rootless-docker.outputs.in-use != 'true'
      run: >
        sudo systemd-run
        --unit=docker-proxy.service
        --description="Bidirectional proxy between rootful and rootless Docker sockets"
        --service-type=exec
        --property=Requires=docker.socket
        --property=PrivateNetwork=true
        --property=PrivateTmp=true
        /lib/systemd/systemd-socket-proxyd "$XDG_RUNTIME_DIR/docker.sock"
      shell: bash

[john-jam]

We also need to have the docker socket configurable when running docker in rootless mode. Allowing to bind a specific socket rather than the proxy socket workaround is a better solution imho. Any plans on updating/merging this PR?

[Kurt-von-Laven]

Yes, the only reason to proxy sockets is that works today. It would be preferable to allow binding to a different socket.

[nathanweeks]

Support for setting the path to the docker socket via DOCKER_HOST environment variable would benefit rootless podman as well (e.g., see this Red Hat support article for allowing rootless podman to be used with docker-compose (v1) by setting DOCKER_HOST: https://access.redhat.com/solutions/7011472)

[Eddman]

Any news on this PR?

We're running ARC + DinD and the official ARC docs are stating that we should change the socket location as it has to be mounted to the second container through an emptyDir volume.

However with hardcoded socket path we can't use actions that require Docker inside containers. The docs also mention setting DOCKER_HOST variable, but this is ignored by the runner.

[nodesocket]

I’m running “containerd” in a self hosted runner, how can I specify a custom path to the socket? Getting an error that /var/run/docker.sock does not exist.

pi@kube-master:~ $ sudo find / -name "docker.sock"

pi@kube-master:~ $ sudo find / -name "containerd.sock"
/run/containerd/containerd.sock

pi@kube-master:~ $ docker info
Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

[BPouw]

For anyone running into this problem, what worked for us is to set a symbolic link in our self hosted runner pointing from the default docker.sock to the user docker.sock

ln -s /home/runner/.docker/run/docker.sock /var/run/docker.sock

Note that this will only work if you have a single user on rootless mode.

[carl-reverb]

As a workaround for running a job container with the GHA runner-scale-set-controller and an autoscalingrunnerset with dind mode configured, I figured out that I could modify the job.<id>.container to mount /run/docker:/run/docker and set DOCKER_HOST=unix:///run/docker/docker.sock.

This enables me to act upon containers, build things, exec and run. However because --network is not supported by GHA, I cannot launch a container with an exposed port from inside my job container and then access that port. For example, I can launch kind from inside the container, but it's useless because the container can't see the exposed port. Still this is a good discovery for some use cases.

For my kind use case I fell back to running the job uncontainerized in the actions-runner pod.

[jcpunk]

This patch would simplify things for me.

[srini-hv]

Hi @aseggemu-yoc , I just updated the MR with the latest main branch. I'm guessing you might have to approve again for the CI to run. Can you take a look?

[paulcostinean]

cc @nikola-jokic Bumping for visibility as it affects all rootless podman/docker users.

@BPouw's solution only allows one linux user to run an action runner and there's no way around it short of LD_PRELOAD hacks.

[davidhjp01]

Bump up, this is also needed for us