Experiencing `permission denied` after v3.1.0 on jobContainers when container user is not 'root'

[fhammerl]

After the rollout actions/checkout v3.1.0, some reported permission denied errors when using the action on a jobContainer running with a container user that is not 'root'. With v3.1.0, we're using commands that write to files to replace deprecated stdout commands.

These are the files in question causing permission issues due to mismatched UIDs between user running the runner process and the user on the container it starts.

Running a container using a container user that is not 'root' is an unsupported path, we've listed some workarounds below.

Summary

In short: the jobContainers in these workflows seem to follow an unsupported path. Namely,

Do not use the USER instruction in your Dockerfile, because you won't be able to access the GITHUB_WORKSPACE

I believe the jobContainer operates as the user 'circleci' in the workflow you linked.

We will roll out the changes again as we haven't seen issues with the officially supported paths.

It is recommended 'not use the USER instruction in your Dockerfile'. In most cases, this means 'run the container as root'.

I understand this introduces some friction, so see below for workarounds and a technical summary.

Detailed Summary

1. In actions/[email protected], we began phasing out the use of some workflow commands that write to stdout with functions that write to temp files instead.
2. If you're running in a jobContainer, these temp files are created by the runner app on your host OS and are owned by the user who started the runner (in case of ubuntu-latest, this user is 'runner' with UID-1001). These files are then mounted to your jobContainer.
3. actions/checkout is a .js file executed using node on your jobContainer. As per the changes in 1., actions/[email protected] attempts to write to these temp files on your jobContainer.

This can have a couple of outcomes:

1. If the jobContainer user is 'root': writes successfully.
2. If the jobContainer user is NOT 'root': a) If the UID of the jobContainer user == UID of host runner user (UID-1001 on ubuntu-latest): write is successful b) If the UID of the jobContainer user != UID of host runner user (UID-1001 on ubuntu-latest): permission denied

The jobContainers in the above workflow seem to run into 2/b.

Fix

As per the docs, don't modify the default user in the Dockerfile. If that's not feasible for you: 👇

Workarounds (unsupported)

• Use actions/[email protected] in your workflow (will be deprecated)
• Match the UID of the host runner user (e.g. UID-1001 on ubuntu-latest today) to the UID of the user on the jobContainer
• Override the default container user and use 'root':

container: 
    image: alpine:latest
    options: --user root

Error logs

##[debug]Running JavaScript Action with default external tool: node16
node:internal/fs/utils:344
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_2[7]

[rGaillard]

We experience the same issue with the v2.5.0 version, our CI is broken, had to rollback :(

[priv-kweihmann]

The same happens with latest release on actions/cache - the common denominator seems to the usage of actions/core 1.10.0. All previous versions that do not use this particular core version are fine

[chenrui333]

Anyone trying to fix this?

[Bo98]

Just brainstorming a bit here: would there be security implications if the runner were to set o+w for file commands? I guess it maybe could for some specific multi-user self-hosted scenarios, but for GitHub-hosted runners and self-hosted runners that opt-in/out: it could make sense and seems like something that could ease concerns here a bit.

Match the UID of the host runner user (e.g. UID-1001 on ubuntu-latest today) to the UID of the user on the jobContainer

I guess the core problem is Docker doesn't really support UID remaps at runtime itself.

Nevertheless, this is a solution I was initially going to chase but the problem is that any mechanism to do so dynamically is disabled in GitHub Actions. While you could rely on a fixed 1001 for GitHub-hosted, it gets trickier with varying self-hosted setups. Initial idea I had was to create an entrypoint that would get the UID of RUNNER_WORKSPACE and use that to change the container user, but unfortuately entrypoints are force disabled unless container hooks are enabled (not entirely sure why), which obviously rules out it working on GitHub-hosted.

[romulus-ai]

Does anyone else have the same security concerns than me if I am forced to runs all Pipeline Stuff and Action under root in a container? I mean, it is a much higher security risk than using an unprivileged user which is just able to execute a bunch of commands it needs.

[paneq]

Just brainstorming a bit here: would there be security implications if the runner were to set o+w for file commands

I like this idea from @Bo98

Match the UID of the host runner user (e.g. UID-1001 on ubuntu-latest today) to the UID of the user on the jobContainer

@fhammerl Is this guaranteed to be a long-term contract that we could rely on in the future with subsequent actions upgrades? Or just one time working working workaround that's still unsupported path?

Can you comment on the technical limitations for supporting container users?

[AngryMaciek]

Is this issue progressing?
I have the same problem as mentioned in https://github.com/actions/checkout/issues/1014 : my test suite requires some tests to be triggered by non-root user; on a self-hosted runner when the workaround is to set user as root in the container they will not work...

[yeikel]

The user id changed between 2.294.0 and 2.303.0 for some reason

In 2.294.0 the user id is 1000 but in 2.303.0 it is 1001

[adminy]

##[debug]Running JavaScript Action with default external tool: node16
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: no such file or directory: unknown
##[debug]Node Action run completed with exit code 126

[theory]

This limitation requiring workflow Docker containers to run as root is requiring contortions to work around (e.g., pgcentralfoundation/pgrx#1652), and to @romulus-ai's point, it seems like a potentially significant security issue. If there is some way to work around it, like using a specific UID and/or username like runner, it would be great to know.

[betimcariad]

@theory I want to point you to my comment which I just created:

https://github.com/actions/checkout/issues/1014#issuecomment-2151943286

Best regards

[theory]

Thanks @betimcariad. Seems like that might work okay until GitHub decides to change the UID and GID. :-(

[betimcariad]

@theory Ah, I was mostly referring to selfhosted runners. I guess, you might be right about Github hosted runners.

Officially, Github anyways says to use root inside the container... https://docs.github.com/en/[email protected]/actions/creating-actions/dockerfile-support-for-github-actions#user