Confused over correct user permissions

[jeffora]

I'm using the docker-compose plugin to run various build steps against a pre-built image from an earlier step. I've repeatedly run into issues with configuring the correct user UID/GID, and was wondering if there was any recommended guidance for this.

Here's the general scenario:

• Separately create a build base image with node and required tools installed. This image configures and uses the node user/group (UID:GID 1000:1000)
• Build a docker image for this service using docker-compose plugin in step 1
• In parallel, run a few tasks using the image from step 1, bind-mounting host directories in to upload artifacts afterwards

The issue I've generally run into here is that bind-mounting a folder from the host to the container doesn't work if the container is run with the node user (1000:1000) as the host user (buildkite-agent:2000:2000) owns the bind-mounted directory and the container user can't write.

In order to fix this, I've specified to run the container with --user 2000:2000. This works for writing to the bind-mounted folder. However, if the container tries to create local directories (in this case, a .cache folder in the home directory), it fails as the 2000 UID does not have a home directory (gets set to /), and does not have write access to either / or $PWD.

Alternatively, I can run the containers as root and I get no permissions errors in the container, but files written to the bind-mounts are owned by root on the host and can't be cleaned up by buildkite-agent, so the build fails to git clean.

Would love any recommendations / tips here. I feel like I must be doing something wrong. I could update the images to have a legitimate user/group with 2000:2000 uid/gid, but that feels like it makes the images very buildkite-specific.

[jeffora]

Some additional context here: I'm running buildkite on the elastic-ci-stack, with EnableDockerUserNamespaceRemap set to false in order to allow multi-stage builds to work (https://github.com/buildkite/elastic-ci-stack-for-aws/issues/392)

[toote]

These have been such common issues that the documentation about running buildkite agents with docker has a specific section about permissions and the issue with user namespace remapping and multi-stage builds is still open in the legacy docker infrastructure (but would appear to work if you are using BuildKit). Not sure there is much we can do in this plugin's code on the matter