moby
Garbage collected credentials not being renewed
Contributing guidelines and issue reporting guide
- [x] I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.
Well-formed report checklist
- [x] I have found a bug that the documentation does not mention anything about my problem
- [x] I have found a bug that there are no open or closed issues that are related to my problem
- [x] I have provided version/information about my environment and done my best to provide a reproducer
Description of bug
Bug description
I am running a regular workload that builds an image against a remote buildkitd and pushes it to a registry that needs authentication. The images build and push fine when buildkitd is first started, but after a few iterations, it starts to consistently fail with the error error: failed to solve: failed to push <my repo address>:<image tag>: server message: insufficient_scope: authorization failed
Looking at the registry logs, I can see error authorizing context: insufficient scope.
The docker config.json that is passed to buildctl on the client does not change - the credentials are static and the build is successful when buildkitd is restarted.
When I run buildctl prune against buildkit, the image is successfully pushed.
My hypothesis is that the credentials are being cached in buildkitd, but they get garbage collected and subsequent builds are not using the credentials from the config in the build - I am unsure how to test this though, as I am not sure how to get the contents of buildkitd's cache.
Reproduction
With a private repo, setup a task that regularly builds and pushes an image from a persistent buildkitd. The command to build the image is:
buildctl \
--addr buildkitd \
build \
--frontend dockerfile.v0 \
--local context=<local dir with Dockerfile and config> \
--local dockerfile=<local dir with Dockerfile and config> \
--output type=image,name=<repository>:<image_tag>,push=true
Version information
$ buildkit --version
buildctl github.com/moby/buildkit v0.22.0 13cf07c97baebd3d5603feecc03f5a46ac98d2a5
$ buildctl debug workers --format json
[{"id":"iki1kq5kztksvbwugphx4uy87","labels":{"org.mobyproject.buildkit.worker.executor":"oci","org.mobyproject.buildkit.worker.hostname":"buildkit-68f9bc76c6-x7czt","org.mobyproject.buildkit.worker.network":"host","org.mobyproject.buildkit.worker.oci.process-mode":"no-sandbox","org.mobyproject.buildkit.worker.selinux.enabled":"false","org.mobyproject.buildkit.worker.snapshotter":"overlayfs"},"platforms":[{"architecture":"amd64","os":"linux"},{"architecture":"amd64","os":"linux","variant":"v2"},{"architecture":"amd64","os":"linux","variant":"v3"},{"architecture":"amd64","os":"linux","variant":"v4"},{"architecture":"386","os":"linux"}],"gcPolicy":[{"all":false,"filter":["type==source.local,type==exec.cachemount,type==source.git.checkout"],"keepDuration":172800000000000,"reservedSpace":0,"maxUsedSpace":512000000,"minFreeSpace":0},{"all":false,"filter":null,"keepDuration":5184000000000000,"reservedSpace":10000000000,"maxUsedSpace":100000000000,"minFreeSpace":25000000000},{"all":false,"filter":null,"keepDuration":0,"reservedSpace":10000000000,"maxUsedSpace":100000000000,"minFreeSpace":25000000000},{"all":true,"filter":null,"keepDuration":0,"reservedSpace":10000000000,"maxUsedSpace":100000000000,"minFreeSpace":25000000000}],"buildkitVersion":{"package":"github.com/moby/buildkit","version":"v0.22.0","revision":"13cf07c97baebd3d5603feecc03f5a46ac98d2a5"},"cdiDevices":null}]
This might be https://github.com/containerd/containerd/pull/11709 . Update to v0.23 . Although that would require multiple sets of credentials (or your registry incorrectly handling cross-repo mount).
What registry are you using? May be some error with the token expiry time reported by the server not being correct or the auth server not returning correct auth scopes.
Its just running Docker Distribution and the same creds are working fine if I restart buildkitd. So I am confident that the creds are still valid (they don't have an exp), and when I restart buildkitd but don't touch the machine that buildctl is running on, or Distribution, it is fine, so my feeling is its something happening in the buildkitd pod
I should add, it is only ever pushing to one repo, and the image that is building from is public, so there is only a single cred.