Self-hosted runner(EKS) Docker Cache Not working(w. efs)

[95jinhong]

Checks

• [X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• [X] I am using charts that are officially provided

Controller Version

0.23.3

Deployment Method

ArgoCD

Checks

• [X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
• [X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Configure and use a self-hosted runner on top of EKS.
2. we want to share and use the cache on Docker Build, so we want to share the /var/lib/docker subfolder with EFS or configure a separate cache folder with EFS and declare the cache in the docker container with cache-from and cache-to options.
3. When I build, it says that the cache folder has no permissions.
4. there is no part in RunnderDeployment Object to declare docker init contianer, so I am not sure how to solve it.

Describe the bug

1. I'm using Self-hosted Runner with Goldenimage with Docker and buildx installed beforehand.
2. to implement Docker Cache, I created /var/lib/docker-cache through EFS and declared the workflow as below.

      - name: Build and Push Image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ env.ECR_URI }}:${{ env.image_tag }}
          build-args: |
            NPM_TOKEN=${{ secrets.READONLY_GPR }}
            BUILD_SERVICE_NAME=${{ env.BUILD_SERVICE_NAME }}
          cache-from: type=local,src=/var/lib/docker-cache
          cache-to: type=local,dest=/var/lib/docker-cache,mode=max

3. However, it says that I don't have permission to the /var/lib/docker-cache folder, as shown below. I'm not sure how to grant that permission.

## workflow log
Run docker/build-push-action@v5
GitHub Actions runtime token ACs
Docker info
Proxy configuration
Buildx version
/usr/local/bin/docker buildx build --build-arg NPM_TOKEN=*** --build-arg BUILD_SERVICE_NAME=api --cache-from type=local,src=/var/lib/docker-cache --cache-to type=local,dest=/var/lib/docker-cache,mode=max --iidfile /runner/_work/_temp/docker-actions-toolkit-uLcqtI/iidfile --provenance mode=min,inline-only=true,builder-id=https://github.com/boostbrothers/backend-ci-test/actions/runs/8044060472 --tag ***.dkr.ecr.ap-northeast-2.amazonaws.com/backend-ci-test/test:54d2[32](https://github.com/boostbrothers/backend-ci-test/actions/runs/8044060472/job/21967103534#step:13:33)49 --metadata-file /runner/_work/_temp/docker-actions-toolkit-uLcqtI/metadata-file --push .
ERROR: mkdir /var/lib/docker-cache: permission denied
Error: buildx failed with: ERROR: mkdir /var/lib/docker-cache: permission denied

Describe the expected behavior

I want the /var/lib/docker container to be used for each Pod (Job) and I want them to share the docker local cache with each other.

Additional Context

My runnerdeployment.yaml looks like this

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: k8s-action-runner-backend-test
  namespace: actions-runner-system
spec:
  template:
    spec:
      ...
      resources:
        requests:
          cpu: 1500m
          memory: 5Gi
      dockerVolumeMounts:
        - mountPath: /var/lib/docker-cache
          name: runner-npm-dockercache-storage
        - mountPath: /var/lib/docker
          name: docker-extra
      volumeMounts:
        - mountPath: /opt/hostedtoolcache
          name: runner-npm-cache-storage
      volumes:
        - name: runner-npm-cache-storage
          persistentVolumeClaim:
            claimName: k8s-action-runner-backend-test-ci
        - name: runner-npm-dockercache-storage
          persistentVolumeClaim:
            claimName: k8s-action-runner-backend-test-ci-dockercache
        - name: docker-extra
          hostPath:
            path: /mnt/docker-extra
            type: DirectoryOrCreate

### Controller Logs

```shell
There doesn't appear to be any special logs.

Runner Pod Logs

## docker container log

time="2024-02-26T05:53:38.304870260Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2024-02-26T05:53:38.304926735Z" level=info msg="containerd successfully booted in 0.038174s"
time="2024-02-26T05:53:54.703885619Z" level=info msg="Loading containers: start."
time="2024-02-26T05:53:54.909627125Z" level=info msg="Loading containers: done."
time="2024-02-26T05:53:54.920842001Z" level=info msg="Docker daemon" commit=311b9ff graphdriver=overlay2 version=24.0.7
time="2024-02-26T05:53:54.921021834Z" level=info msg="Daemon has completed initialization"
time="2024-02-26T05:53:54.947228154Z" level=info msg="API listen on /run/docker.sock"

time="2024-02-26T06:09:58.167664345Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-02-26T06:09:58.167730410Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-02-26T06:09:58.168723970Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-02-26T06:09:58.168780188Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-02-26T06:10:00.636146598Z" level=info msg="ignoring event" container=8e7f9e894c12ff796b51ec341e39091b490aeeb3409f261f0e1ab1afa970056b module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2024-02-26T06:10:00.636124578Z" level=info msg="shim disconnected" id=8e7f9e894c12ff796b51ec341e39091b490aeeb3409f261f0e1ab1afa970056b namespace=moby
time="2024-02-26T06:10:00.636237278Z" level=warning msg="cleaning up after shim disconnected" id=8e7f9e894c12ff796b51ec341e39091b490aeeb3409f261f0e1ab1afa970056b namespace=moby
time="2024-02-26T06:10:00.636253531Z" level=info msg="cleaning up dead shim" namespace=moby
Prestop hook started
Waiting for dockerd to start
15
Prestop hook stopped
time="2024-02-26T06:10:14.209947178Z" level=info msg="Processing signal 'terminated'"
time="2024-02-26T06:10:14.211190041Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2024-02-26T06:10:14.211758404Z" level=info msg="Daemon shutdown complete"
time="2024-02-26T06:10:14.211835623Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2024-02-26T06:10:14.211855654Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
rpc error: code = NotFound desc = an error occurred when try to find container "2a7e0e4509fa8609f62baf7f753337dc169841eb7afe41b0bd5de00d545decdd": not found

runner container log

√ Connected to GitHub

Current runner version: '2.313.0'
2024-02-26 05:54:20Z: Listening for Jobs
2024-02-26 06:09:28Z: Running job: api / build
2024-02-26 06:10:12Z: Job api / build completed with result: Failed
√ Removed .credentials
√ Removed .runner
Runner listener exit with 0 return code, stop the service, no retry needed.
Exiting runner...
2024-02-26 06:10:12.912  NOTICE --- Runner init exited. Exiting this process with code 0 so that the container and the pod is GC'ed Kubernetes soon.

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[AlonAvrahami]

https://github.com/actions/actions-runner-controller/issues/3244#issuecomment-1906634729

[iliasnaamane2]

Hello, I have the same need as you and I want the same result. I have tried your solution and I have faced the same error because it tries to create the folder in the runner container but the volume is mounted in the docker container of the runner pod. Can someone help out how to enable docker layer caching? Thanks