garden-io
`cluster-buildkit` errors when attempting to run on GKE Autopilot
Bug
The cluster-buildkit build mode errors when attempting to run on GKE Autopilot in either rootless or rootful mode. Together with https://github.com/garden-io/garden/issues/3469 it appears in-cluster image building is currently blocked from Autopilot clusters. As Autopilot is now the recommended way of running GKE, we can expect more users to encounter this issue as they onboard to Garden.
Current Behavior
In rootless mode, Garden errors with:
error: failed to solve: failed to read dockerfile: failed to mount
/home/user/.local/tmp/buildkit-mount4232801934: [{Type:bind
Source:/home/user/.local/share/buildkit/runc-overlayfs/snapshots/snapshots/2/fs Options:[rbind ro]}]:
operation not permitted
And if you're running an Autopilot cluster with root_ful_ BuildKit, this is also not supported:
Got error from Kubernetes API (replaceNamespacedDeployment) - admission webhook
"gkepolicy.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it
violates one or more constraints.
Violations details: {"[denied by autogke-disallow-privilege]":["container buildkitd is privileged; not
allowed in Autopilot"]}
Expected behavior
Rootless BuildKit should succeed.
Reproducible example
Use my https://gist.github.com/worldofgeese/d6c9b913637cf6998e44b77c44cce73f with cluster-buildkit uncommented and kaniko commented. Be sure your cluster is authorized against your chosen image registry.
Workaround
Use local build mode.
Suggested solution(s)
Autopilot will likely never allow privileged containers. For cluster-buildkit to run rootlessly it looks like a fix is to create an emptyDir at /home/user/.local/share/buildkit. See https://github.com/moby/buildkit/issues/879#issuecomment-1240347038 for more.
Your environment
- OS: Windows 11 with WSL
- How I'm running Kubernetes: GKE Autopilot
garden version
0.12.48
