[Amazon Linux 2] docker buildx create & build error (rootless): `mount sysfs:/sys (via /proc/self/fd/6), flags: 0xe: operation not permitted: unknown`

[Rez0k]

Hi,

I am running docker in docker rootless (dind-rootless) on kubernetes cluster (eks cluster with containerd as container runtime). everything is working on dind, but when migrating to dind-rootless I am getting errors while running the commands:

docker buildx create --name ci-builder --config buildkitd.toml --use --driver-opt image=moby/buildkit:v0.10.4-rootless --buildkitd-flags '--security-opt seccomp=unconfined --security-opt apparmor=unconfined --device /dev/fuse --oci-worker-no-process-sandbox --allow-insecure-entitlement security.insecure'

docker buildx build --push --builder ci-builder

[+] Building 0.2s (1/1) FINISHED
=> ERROR [internal] booting buildkit 0.2s => => starting container buildx_buildkit_ci-builder0 0.2s

[internal] booting buildkit:

---

ERROR: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "sysfs" to rootfs at "/sys": mount sysfs:/sys (via /proc/self/fd/6), flags: 0xe: operation not permitted: unknown

additional info: docker info -

Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.9.1)

Server: Containers: 2 Running: 0 Paused: 0 Stopped: 2 Images: 2 Server Version: 20.10.18 Storage Driver: fuse-overlayfs Logging Driver: json-file Cgroup Driver: none Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6 runc version: v1.1.4-0-g5fd4c4d1 init version: de40ad0 Security Options: seccomp Profile: default rootless Kernel Version: 5.4.209-116.367.amzn2.x86_64 Operating System: Alpine Linux v3.16 (containerized) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.482GiB Name: jenkins-agent-st3p2 ID: EWAZ:EPLH:6FZS:777F:BT5R:JYGV:PW42:WIBV:7USU:4D6H:FRVC:U4D7 Docker Root Dir: /home/rootless/.local/share/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: docker-registry.docker-registry:5000 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine

WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.

I tried running the dind-rootless with the fuse-overlayfs storage driver and nothing changed. I also tried modifying the buildkitd-flags and nothing changed.

Thanks !

[AkihiroSuda]

Is this Amazon EKS?

[Rez0k]

Is this Amazon EKS?

@AkihiroSuda yes

[Lavaerius]

I'm also seeing this problem, but on AKS

[lpfann]

We also had the same issue. After debugging a bit what helped for us deactivating Istio namespace injection.

[AkihiroSuda]

Specifying --oci-worker-no-process-sandbox may work (although it seems already specified for the OP's case)

• https://github.com/moby/buildkit/pull/4075

[AKrekhovetskyi]

Hey there! I also have similar issue in IDX when I run docker compose up. Here is the output of the docker version command

Client:
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.21.11
 Git commit:        v24.0.9
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.21.11
  Git commit:       v24.0.9
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        v1.7.13
 runc:
  Version:          1.1.12
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit3862372010
 slirp4netns:
  Version:          1.2.2
  GitCommit:        0ee2d87523e906518d34a6b423271e4826f71faf

And docker compose version

Docker Compose version 2.23.1