GKE read-only host filesystem problem

[sj26]

There's a problem when running the buildkite agent on GKE as a container when you also want to use docker-based builds. The agent container must share a volume into the docker containers which run commands. This can't be done directly, it has to be done vicariously via the host filesystem. But the host filesystem is read-only by default, except for certain paths. So the paths must be configured to a writeable path on the host, then those paths must be mounted into the agent container as a volume mount.

Here's a reply about this I just sent to a customer in support, which might be useful as a base for an addition to our agent installation docs on Google Cloud / GKE:

I notice you're running the agent on GKE. Is the agent itself running in a docker container?

This junit plugin uses docker to run a ruby container to compose the annotation. It mounts the plugin directory into the docker container as a volume to do so. But docker volumes are based on the host filesystem, so if the agent is running within a container itself then it's trying to map a container filesystem into another container filesystem instead, which won't work.

You can fix this by mapping a matching host filesystem path into the buildkite agent container. The host filesystem path must be the same as the container filesystem path because there's it's way too hard to rewrite paths in between the agent container and the host.

The next problem is that GKE hosts have a read-only filesystem. See here for more details:

https://cloud.google.com/container-optimized-os/docs/concepts/disks-and-filesystem

The buildkite agent docker image uses "/buildkite" as default for builds, hooks, and plugins, which isn't going to work if mapped as-is to a GKE host — it won't be writeable. But you can modify where the agent wants to put these things with environment variables, per: https://buildkite.com/docs/agent/v3/configuration

We've found that using "/var/buildkite" is a good option for GKE — it will be writeable on the host. But the config file is also in there at /buildkite/buildkite-agent.cfg, so you don't want to override the whole thing, just the path for builds and plugins. So you can modify your kubernetes container spec to do something like:

spec:
  containers:
  - name: buildkite-agent
    image: buildkite/agent
    env:
    - name: BUILDKITE_BUILD_PATH
      value: /var/buildkite/builds
    - name: BUILDKITE_PLUGINS_PATH
      value: /var/buildkite/plugins
    volumeMounts:
    - name: buildkite
      mountPath: /var/buildkite
  volumes:
  - name: buildkite
    hostPath: {path: /var/buildkite}

I'm sorry if these exact instructions don't work, I don't use GKE very much, but the general problem is that the plugin is using a Docker container, including a volume, and that volume needs to mapped from the agent's filesystem to the container's filesystem.