runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

Cannot write to `$GITHUB_OUTPUT` file from Docker action using user with UID 1000

Open vorporeal opened this issue 1 year ago • 7 comments

Description

We have a custom action that spins up an Arch Linux Docker container and runs a shell script within it that writes to $GITHUB_OUTPUT. The Dockerfile creates a user named build with UID 1000, and invokes USER build so entrypoint.sh executes as this non-root user. (We cannot run the script we need to as root.)

This has been working fine up until today, where it started failing. It failed when running with the 20241103.1.0 runner image (ubuntu-20.04) but passed a few hours earlier with the 20241006.1.0 runner image.

We're getting a "Permission denied" error when attempting to append to $GITHUB_OUTPUT. I'm guessing the user owning the $GITHUB_OUTPUT file outside of the container is no longer assigned UID 1000 and/or the permissions on the file don't grant all users write permissions, leading to this issue.

Platforms affected

  • [ ] Azure DevOps
  • [ ] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [x] Ubuntu 20.04
  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] macOS 12
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [ ] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022

Image version and build link

Image: ubuntu-20.04 Version: 20241103.1.0

Screenshot of the failing step:

image

Output copied below, in case it's useful to have it in text form:

Run ./.github/actions/bundle_arch_package
Building docker image
/usr/bin/docker run --name c6dc350e4cd7f005b529aff3e296d6904d9d0_a09fdb --label 5c6dc3 --workdir /github/workspace --rm -e "CARGO_TERM_COLOR" -e "CONFIG_FILE" -e "CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE" -e "GOOGLE_APPLICATION_CREDENTIALS" -e "GOOGLE_GHA_CREDS_PATH" -e "CLOUDSDK_CORE_PROJECT" -e "CLOUDSDK_PROJECT" -e "GCLOUD_PROJECT" -e "GCP_PROJECT" -e "GOOGLE_CLOUD_PROJECT" -e "INPUT_CHANNEL" -e "INPUT_RELEASE-TAG" -e "INPUT_ARCH" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/warp-internal/warp-internal":"/github/workspace" 5c6dc3:50e4cd7f005b529aff3e296d6904d9d0  "dev" "v0.2024.11.06.16.24.dev_00" "aarch64"
Setting release tag to v0.2024.11.06.16.24.dev_00
Setting architecture to aarch64
Skipping `cargo build` step due to --skip-build argument
./script/linux/bundle: line 195: /github/file_commands/set_output_2b1cf403-5[17](https://github.com/warpdotdev/warp-internal/actions/runs/11707713664/job/32608800523#step:11:18)9-4f32-ba69-1b387703b018: Permission denied

Is it regression?

Yes, it worked on 20241006.1.0 a few hours earlier

Expected behavior

We should be able to write to the file path at $GITHUB_OUTPUT from within a Docker container action without errors.

Actual behavior

The write fails with "Permission denied".

Repro steps

I have not explicitly tested this, but my expectation is that the following would reproduce the issue (it's a simplification of our failing action):

Define a Docker action with the following Dockerfile:

# Explicitly use an older version, as it appears that the next release
# (base-devel-20240714.0.246936) causes fakeroot (invoked by makepkg)
# to hang.
FROM archlinux:base-devel-20240101.0.204074

ARG USERNAME=build
ARG USER_UID=1000
ARG USER_GID=$USER_UID

# Create our build user.
RUN groupadd --gid $USER_GID $USERNAME \
    && useradd --uid $USER_UID --gid $USER_GID -m $USERNAME

# Run as our build user instead of root, as makepkg must be run as a non-root
# user.
USER $USERNAME

COPY entrypoint.sh /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Define the following entrypoint.sh file:

#!/usr/bin/env bash

echo "test=value" >> "$GITHUB_OUTPUT"

Use that action in a workflow running on ubuntu-20.04 that uses the new runner image.

vorporeal avatar Nov 06 '24 17:11 vorporeal

We had the same exact issue. We have more or less the same Dockerfile strategy.

Issue came when they changed the default version of Ubuntu image to 20241103.1.0 from 20241015.1.0. Runner could not read files shared in a volume between the runner and the container instance.

I saw that Buildx changed between version. I tried to revert to 0.17.1, but at last it didn't work.

Somehow someone reverted something, now our CI runs 20241015.1.0 again.

Hope this help a little.

VincentTanguayCasgrain avatar Nov 06 '24 20:11 VincentTanguayCasgrain

Hi @vorporeal - we're looking into this issue , we will update on it ASAP. thank you !

kishorekumar-anchala avatar Nov 07 '24 08:11 kishorekumar-anchala

Hi @vorporeal - For above mentioned issue, -Firstly, you can try to set permissions on $GITHUB_OUTPUT before entering the container. You can do this by running a chmod command to make the file writable by other users (including UID 1000) before starting the Docker container.

-Further, Inside your Docker action (entrypoint.sh), you can write to a custom file instead of $GITHUB_OUTPUT then append it to $GITHUB_OUTPUT in a separate step as:-

-name: Append Docker output to $GITHUB_OUTPUT
  run: cat /github/workspace/custom_output >> $GITHUB_OUTPUT

Feel free to update us on this after trying above . In mean time, we are checking more into this and will update you, Thanks!

Prabhatkumar59 avatar Nov 07 '24 13:11 Prabhatkumar59

Hi @Prabhatkumar59 and @kishorekumar-anchala - looks like this is biting us again with the 20241209 image release.

If it's reasonable and secure to set a+rw on $GITHUB_OUTPUT, is there a reason that isn't done for the entire working folder (and relevant files like $GITHUB_OUTPUT)?

Working around the $GITHUB_OUTPUT issue leads to us failing to write other files into the working directory, e.g.:

mkdir: cannot create directory ‘/github/workspace/target/dist’: Permission denied

If there's no reason for /github/workspace to not be a+rw, would be nice if that were set up in the runner image so that this works out-of-the-box for all users of GitHub Actions.

vorporeal avatar Dec 13 '24 17:12 vorporeal

I printed out the contents of /etc/passwd from a ubuntu-latest runner:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-network:x:100:102:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:101:103:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
messagebus:x:102:105::/nonexistent:/usr/sbin/nologin
systemd-timesync:x:103:106:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
syslog:x:104:111::/home/syslog:/usr/sbin/nologin
_apt:x:105:65534::/nonexistent:/usr/sbin/nologin
tss:x:106:112:TPM software stack,,,:/var/lib/tpm:/bin/false
uuidd:x:107:113::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:114::/nonexistent:/usr/sbin/nologin
sshd:x:109:65534::/run/sshd:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
landscape:x:111:1[16](https://github.com/warpdotdev/warp-internal/actions/runs/12322467661/job/34396094594?pr=12671#step:2:17)::/var/lib/landscape:/usr/sbin/nologin
fwupd-refresh:x:112:1[17](https://github.com/warpdotdev/warp-internal/actions/runs/12322467661/job/34396094594?pr=12671#step:2:18):fwupd-refresh user,,,:/run/systemd:/usr/sbin/nologin
_chrony:x:113:122:Chrony daemon,,,:/var/lib/chrony:/usr/sbin/nologin
lxd:x:999:100::/var/snap/lxd/common/lxd:/bin/false
sphinxsearch:x:114:123:Sphinx fulltext search service,,,:/var/run/sphinxsearch:/usr/sbin/nologin
systemd-coredump:x:115:124:systemd Core Dumper,,,:/run/systemd:/usr/sbin/nologin
dnsmasq:x:116:65534:dnsmasq,,,:/var/lib/misc:/usr/sbin/nologin
mysql:x:117:126:MySQL Server,,,:/nonexistent:/bin/false
postgres:x:1[18](https://github.com/warpdotdev/warp-internal/actions/runs/12322467661/job/34396094594?pr=12671#step:2:19):127:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash
runneradmin:x:1000:1000:Ubuntu:/home/runneradmin:/bin/bash
runner:x:1001:128:,,,:/home/runner:/bin/bash

I notice that runneradmin is a user in there, assigned UID 1000.

Any chance you all added a new user to the runner images in a way that changed the UID of the user that is actually running the scripts? If so, would you mind making runneradmin UID 1001 and runner UID 1000 to avoid breaking every action and workflow that may expect to be using UID 1000?

vorporeal avatar Dec 13 '24 20:12 vorporeal

As a slight aside, it is completely unacceptable that GitHub does not provide any way of either 1) testing new runner images before they are rolled out OR 2) temporarily rolling back to the previous image if the new one breaks your workflows.

We cannot prepare for the release of a new image, and if the new image breaks us, we must fix-forward and cannot quickly mitigate any issues by using the old version while we figure out what the hell broke.

We have been unable to push out an important update to our product today due to this breaking our release workflow, and there is no way to quickly remediate.

vorporeal avatar Dec 13 '24 20:12 vorporeal

Hi @vorporeal - Really thanks for your input! We are closely checking into the details provided above https://github.com/actions/runner-images/issues/10915#issuecomment-2542259255 and will update you soon.

Prabhatkumar59 avatar Dec 17 '24 10:12 Prabhatkumar59

Hi @vorporeal - Could you please confirm us once whether above issue is resolved or not? Appreciate your inputs, thanks:)

Prabhatkumar59 avatar Feb 17 '25 11:02 Prabhatkumar59

FYI the only change of UID from 1000 → 1001 was for large runners:

  • https://github.com/actions/runner-images/issues/10936

The announcement would make it sound like for standard runners, the users were always like this? 🤷

(The issue above includes tips about grabbing the current UID via arguments for more flexibility…)

janbrasna avatar Feb 18 '25 10:02 janbrasna

Hi @janbrasna - Thanks for your comment. We will check and update on this shortly.

Prabhatkumar59 avatar Feb 20 '25 07:02 Prabhatkumar59

Hi @vorporeal - The changes have been already reverted for the above issue, so we’re closing this for now. Please let us know if you encounter any further issues. Thank you!

Prabhatkumar59 avatar Feb 21 '25 15:02 Prabhatkumar59