moby
Rootless buildkit does not accept configuration file
Contributing guidelines and issue reporting guide
- [x] I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.
Well-formed report checklist
- [x] I have found a bug that the documentation does not mention anything about my problem
- [x] I have found a bug that there are no open or closed issues that are related to my problem
- [x] I have provided version/information about my environment and done my best to provide a reproducer
Description of bug
Bug description
I am trying to create buildkit base image, to use it for building docker images within our pipelines. Buildkit is supposed to run in unprivileged mode as a Kubernetes pod. One of the things we want to achieve is to use GCP mirror instead of Docker registry for pulling down images.
Reproduction
As per documentation, toml file should be placed in this path "~/.config/buildkit/buildkitd.toml": https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md
This is my Dockerfile
FROM moby/buildkit:rootless
USER root
RUN mkdir -p /home/user/.config/buildkit
COPY buildkit/buildkitd.toml /home/user/.config/buildkit/buildkitd.toml
RUN apk update && apk add --no-cache bash curl python3 tar libc6-compat
RUN curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-linux-x86_64.tar.gz \
&& tar -xf google-cloud-cli-linux-x86_64.tar.gz \
&& ./google-cloud-sdk/install.sh --quiet \
&& rm google-cloud-cli-linux-x86_64.tar.gz \
&& chown -R user:user /home/user/
ENV PATH="/google-cloud-sdk/bin:${PATH}"
USER user
And this is TOML
trace = true
debug = true
[registry."docker.io"]
mirrors = ["mirror.gcr.io"]
So when I run buildctl-daemonless.sh build from the output it seems its pulling from docker.io. This is command I use
buildctl-daemonless.sh build --frontend dockerfile.v0 --local context=/images --local dockerfile=cloudcustodian --opt filename=python3.8-slim.Dockerfile --output type=image,name=cloudcustodian:python3.8-slim,push=false
Version information
buildctl-daemonless.sh -version
buildctl github.com/moby/buildkit v0.23.1 0a230574721405f79ff7361596ec55045f3685bc
It's seems like it's pulling from docker.io, but actually it's pulling from your mirror registry. I have the same output but I know exactly that buildkit:rootless is using mirror because of base image in our registry doesn't exists in docker.io.
I am not sure if that is true, I cannot verify it in my case. I have tried to scramble the URL of mirror to be "mirrorzzz.gcr.io" and its pulling without any issues, so I thought it might be just its ignoring the mirror since image is on docker.io.
I have tried to set this:
trace = true debug = true [registry."docker.io"] mirrors = ["mcr.microsoft.com"]
To make microsoft registry as a "mirror", and to try to pull image that is only on microsoft registry and it fails:
buildctl-daemonless.sh build --frontend dockerfile.v0 --local context=/images --local dockerfile=cloudcustodian --opt filename=python3.8-slim.Dockerfile --output type=image,name=cloudcustodian:python3.8-slim,push=false
[+] Building 1.5s (2/2) FINISHED
=> [internal] load build definition from python3.8-slim.Dockerfile 0.1s
=> => transferring dockerfile: 664B 0.0s
=> ERROR [internal] load metadata for docker.io/library/playwright:v1.47.2-jammy
To make microsoft registry as a "mirror", and to try to pull image that is only on microsoft registry and it fails:
How does that image exist in that registry?
» docker buildx imagetools inspect mcr.microsoft.com/library/playwright:v1.47.2-jammy
ERROR: mcr.microsoft.com/library/playwright:v1.47.2-jammy: not found
If you want to just check that your configuration file gets loaded, then change some other property, eg. gc, and see if you see that config in buildctl debug workers --verbose.
hi @tonistiigi , thanks for info provided. Regarding microsoft image, its "mcr.microsoft.com/playwright:v1.47.2-jammy" it does not have /library in the name.
Regarding debug command, since this is rootless image, it uses some wrapper script to start temporary "daemon", by default buildctl debug workers --verbose wont work:
buildctl debug workers --verbose
error: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing: dial unix /run/user/1000/buildkit/buildkitd.sock: connect: no such file or directory"
But it works this way:
$ buildctl-daemonless.sh debug workers --verbose
ID: t3812hbufdu9cx8gr7ezhn4oa
Platforms: linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6
BuildKit: github.com/moby/buildkit v0.23.1 0a230574721405f79ff7361596ec55045f3685bc
Labels:
org.mobyproject.buildkit.worker.executor: oci
org.mobyproject.buildkit.worker.hostname: 988c53a7e42f
org.mobyproject.buildkit.worker.network: host
org.mobyproject.buildkit.worker.oci.process-mode: sandbox
org.mobyproject.buildkit.worker.selinux.enabled: false
org.mobyproject.buildkit.worker.snapshotter: overlayfs
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep duration: 48h0m0s
Maximum used space: 512MB
GC Policy rule#1:
All: false
Keep duration: 1440h0m0s
Reserved space: 10GB
Minimum free space: 202GB
Maximum used space: 100GB
GC Policy rule#2:
All: false
Reserved space: 10GB
Minimum free space: 202GB
Maximum used space: 100GB
GC Policy rule#3:
All: true
Reserved space: 10GB
Minimum free space: 202GB
Maximum used space: 100GB
I don't see anything configfile related here
@jovantanasic From my own experience:
- buildkit will automatically fall back to docker.io if none of the mirrors work (there is no way to disable this)
- to tell for sure where it pulled from, enable debug logging and look at the resulting buildkitd logs (NOT the buildctl logs)
@rittneje thanks for feedback, in my case bottom line is that I want to find a way to pass configuration file to rootless buildkit. Setting mirror is just a first part of configuration I want to use.
Regarding microsoft image, its "mcr.microsoft.com/playwright:v1.47.2-jammy" it does not have /library in the name.
Then you can't use it as a mirror for docker hub. All images in Hub have org in their path.
I don't see anything configfile related here
Did you disable GC in config and expect to see different output?
@tonistiigi I have the exactly the same problem with buildkit 0.23.1 (not sure whether it works on other versions though). I have a buildkitd.toml that looks like this
UPDATE: well, this is really weird. If the <IP address> in the config bellow is localhost or 127.0.0.1 it works, if I specify a proxy (exactly the same configuration!) that is run on another machine in the same sub-net -- it doesn't! I run curl from the builder container and I can reach both proxies without a problem. But the worst thing is that buildkit silently ignores mirrors without saying anything in the log
debug = true
[registry."docker.io"]
mirrors = ["<IP address>:5000"]
insecure = true
http = true
I'm creating a container like this
docker --context default container run -d --name linuxkit-builder \
--privileged \
-v $(pwd)/buildkit.toml:/etc/buildkit/buildkitd.toml \
--network=host \
moby/buildkit:$(BUILD_KIT_VERSION) \
--allow-insecure-entitlement network.host \
--config /etc/buildkit/buildkitd.toml \
--debug --addr tcp://0.0.0.0:1234; \
I know that /etc/buildkit/buildkitd.toml is beeing parsed because I see errors in the logs if I put some incorrect options into it so it is not ignored. However [registry."docker.io"] is ignored. I do not see any access from buildkit in the logs of my geristry proxy
@rucoder I could not reproduce this. It is clearly making the request to the mirror first.
@rucoder Is the problem in your case that you are setting the insecure properties to wrong host? If you want to to mirror Docker Hub (TLS) with a local plain-http mirror then the confi would be something like.
debug = true
[registry."docker.io"]
mirrors = ["192.168.215.5:5000"]
[registry."192.168.215.5:5000"]
insecure = false
http = true
@rucoder Is the problem in your case that you are setting the insecure properties to wrong host? If you want to to mirror Docker Hub (TLS) with a local plain-http mirror then the confi would be something like.
debug = true [registry."docker.io"] mirrors = ["192.168.215.5:5000"] [registry."192.168.215.5:5000"] insecure = false http = true
@tonistiigi thanks a lot!! this works, but this is soooo not intuitive . besides why it works if I run proxy on localhost with only
[registry."docker.io"]
mirrors = ["localhost:5000"]
thanks a lot!! this works, but this is soooo not intuitive
If you think about it, then in any other way it would be impossible to add an insecure mirror to a secure registry.
besides why it works if I run proxy on localhost with only
Because, localhost is a special case that allows plainhttp by default https://github.com/containerd/containerd/blob/main/core/remotes/docker/resolver.go#L195
We're running into the same issue after following GitLab's guide on how to set up BuildKit to use a registry mirror (link).
Even after enabling the debug and trace options, it seems like the custom BuildKit configuration file is ignored or not loaded.
cat <<EOF > /tmp/buildkit.toml
debug = true
trace = true
[log]
format = "json"
[registry."docker.io"]
mirrors = ["harbor.example.com/docker.io"]
EOF
BUILDKITD_FLAGS: --oci-worker-no-process-sandbox --config /tmp/buildkit.toml
