buildkit
buildkit copied to clipboard
moby
Secret mount errors with permission denied when using userns-remap
Hi,
I'm having a problem remarkably similar to #1196 that I get a permission denied error when trying to use BuildKit secrets and user namespace remapping.
Docker version: 20.10.6 (Ubuntu 18.04) and several others - some creative bisecting showed this worked in 19.03.6 (after #1196 was fixed) up to 19.03.14, but didn't work in 19.03.15 and still doesn't work in 20.10.6. So maybe a change in Buildkit 0.8.0?
Dockerfile:
# syntax=docker/dockerfile:1.2
FROM ubuntu:18.04
RUN --mount=type=secret,id=mysecretsecret cat /run/secrets/mysecret
/etc/docker/daemon.json (it's not actually running in Jenkins, that's just where the automation that built this VM came from):
{
"log-driver": "journald",
"userns-remap": "jenkins"
}
Build command:
echo "secret" > /tmp/secret.txt
DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain --secret id=mysecret,src=/tmp/secret.txt .
Output:
jenkins@instance:~/build$ DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain --secret id=mysecret,src=/tmp/secret.txt .
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 37B 0.0s done
#2 DONE 0.1s
#1 [internal] load .dockerignore
#1 transferring context: 2B 0.0s done
#1 DONE 0.1s
#3 resolve image config for docker.io/docker/dockerfile:1.2
#3 DONE 1.3s
#4 docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b...
#4 CACHED
#5 [internal] load build definition from Dockerfile
#5 transferring dockerfile: 37B done
#5 DONE 0.0s
#6 [internal] load metadata for docker.io/library/ubuntu:18.04
#6 DONE 0.8s
#7 [1/2] FROM docker.io/library/ubuntu:18.04@sha256:85c0fa27a9e174e31b54b61...
#7 resolve docker.io/library/ubuntu:18.04@sha256:85c0fa27a9e174e31b54b61858a95ca5c36a73c0e9d85a9484b6bd282219b083 done
#7 CACHED
#8 [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 0.251 container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting "/var/lib/docker/998.998/tmp/buildkit-secrets508950687/ryf2i7j13k1qj2isonhp5rjy5" to rootfs at "/run/secrets/mysecret" caused: stat /var/lib/docker/998.998/tmp/buildkit-secrets508950687/ryf2i7j13k1qj2isonhp5rjy5: permission denied
#8 ERROR: executor failed running [/bin/sh -c cat /run/secrets/mysecret]: runc did not terminate sucessfully
------
> [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c cat /run/secrets/mysecret]: runc did not terminate sucessfully
Hope that helps, let me know if there's any other info that would be useful
Not sure if this helps your particular workload or not. but one of the new feature introduced in 20.10.0 is you can mount secrets from environment variables.
MY_SECRET_ENV=secret \
DOCKER_BUILDKIT=1 \
docker build --no-cache --progress=plain --secret id=mysecret,env=MY_SECRET_ENV .
in the Dockerfile, you should be able to mount the secret in the exact same way.
FROM ubuntu:18.04
RUN --mount=type=secret,id=mysecretsecret cat /run/secrets/mysecret
you can find more detail about this under the build section of the release note here https://docs.docker.com/engine/release-notes for version 20.10.0
The suggestion about mounting secrets from environment variables doesn't work for me :disappointed:
Distributor ID: Ubuntu
Description: Ubuntu 21.10
Release: 21.10
Codename: impish
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:33 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:41 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Looks like this issue has been fixed in the latest docker release 20.10.15 through an upgrade of runc from 1.0.3 to 1.1.0. I haven't had time to investigate in detail yet. However, I suspect this change fixed the issue https://github.com/opencontainers/runc/pull/3057.
For others experiencing the same issue. Maybe try to test again after upgrading docker engine to 20.10.15
Did the upgrade resolve this for people? I ask because I'm running 20.10.23 in my CI/CD platform and I'm running into this problem.
This script:
docker version
id -u
export DOCKER_BUILDKIT=1
export MYSECRET=tomwashere
dockerfile="
FROM alpine:latest
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
"
(
set -x
echo "$dockerfile" | docker build --no-cache --progress=plain --secret id=mysecret,env=MYSECRET -
)
Show the permission error:
Client: Docker Engine - Community
Version: 20.10.23
API version: 1.41
Go version: go1.18.10
Git commit: 7155243
Built: Thu Jan 19 17:36:21 2023
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.23
API version: 1.41 (minimum version 1.12)
Go version: go1.18.10
Git commit: 6051f14
Built: Thu Jan 19 17:34:26 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
6790
+ echo '
FROM alpine:latest
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
'
+ docker build --no-cache --progress=plain --secret id=mysecret,env=MYSECRET -
#1 [internal] load build definition from Dockerfile
#1 sha256:c4993d2cc6263849afed878d861540739484a7e3c661baa4f3eaee71123c6a85
#1 transferring dockerfile: 183B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:8db09824d0eb806bfc536260cd837024bc0f5f8035225921bf1c33d87238313b
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/alpine:latest
#3 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#3 DONE 0.0s
#4 [1/2] FROM docker.io/library/alpine:latest
#4 sha256:665ba8b2cdc0cb0200e2a42a6b3c0f8f684089f4cd1b81494fbb9805879120f7
#4 CACHED
#5 [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#5 sha256:75601a522ebe80ada66dedd9dd86772ca932d30d7e1b11bba94c04aa55c237de
#5 0.088 container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting "/var/lib/docker/6790.1000/tmp/buildkit-secrets418852461/158awosrg1w12fk0l4zhnodwk" to rootfs at "/run/secrets/mysecret" caused: stat /var/lib/docker/6790.1000/tmp/buildkit-secrets418852461/158awosrg1w12fk0l4zhnodwk: permission denied
#5 ERROR: executor failed running [/bin/sh -c cat /run/secrets/mysecret]: exit code: 1
------
> [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret:
------
executor failed running [/bin/sh -c cat /run/secrets/mysecret]: exit code: 1
The uid of my userns-remap is 6790.
Running the following Dockerfile
FROM ubuntu:18.04
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
With
echo "secret" > /tmp/secret.txt
DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain --secret id=mysecret,src=/tmp/secret.txt .
results in
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 148B done
#1 DONE 0.0s
#2 resolve image config for docker-image://docker.io/docker/dockerfile:1.2
#2 DONE 0.6s
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 148B done
#1 DONE 0.0s
#3 docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc
#3 CACHED
#4 [internal] load build definition from Dockerfile
#4 DONE 0.0s
#5 [internal] load .dockerignore
#5 transferring context: 35B done
#5 DONE 0.0s
#6 [auth] library/ubuntu:pull token for registry-1.docker.io
#6 DONE 0.0s
#7 [internal] load metadata for docker.io/library/ubuntu:18.04
#7 DONE 1.4s
#8 [1/2] FROM docker.io/library/ubuntu:18.04@sha256:152dc042452c496007f07ca9127571cb9c29697f42acbfad72324b2bb2e43c98
#8 CACHED
#9 [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#9 0.228 secret
#9 DONE 0.2s
#10 exporting to image
#10 exporting layers done
#10 writing image sha256:96d6ae151b29780b1757c922e5118604414f0d29b9c8dffc420a6bb308e7b529 done
#10 DONE 0.0s
Running docker version results with output
Client: Docker Engine - Community
Version: 26.1.2
API version: 1.45
Go version: go1.21.10
Git commit: 211e74b
Built: Wed May 8 13:59:59 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.2
API version: 1.45 (minimum version 1.24)
Go version: go1.21.10
Git commit: ef1912d
Built: Wed May 8 13:59:59 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Only difference i made was in Dockerfile changing id from id=mysecretsecret to id=mysecret
Hope this adds some info that could be helpful in solving this issue.
But it looks like there is no issue for me
