HOME is overridden for containers

[andreineculau]

Describe the bug If passing the HOME environment variable to a container configuration, it will get ignored and set to /github/home anyway as per https://github.com/actions/runner/blob/4e85b8f3b7493aa1adf9dd0a574a9fa2bf1add71/src/Runner.Worker/ContainerOperationProvider.cs#L258

To Reproduce

A workflow with

jobs:
  main:
    runs-on: ubuntu-latest
    container:
      image: 'rokmoln/sf-ubuntu-bionic-common:0.2.80'
      options: '--user sf:sf' # this user's home is /home/sf
      env:
        HOME: /home/sf # tried to change HOME from /github/home to /home/sf. doesn't work
    steps:
      - shell: bash
        run: |
          echo $HOME

will echo /github/home instead of /home/sf.

Expected behavior Expected HOME to be left as is, as sf user's HOME, or at the very least to be allowed to enforce it via the env dictionary.

Runner Version and Platform

2.274.2

What's not working?

N/A

Job Log Output

N/A

Runner and Worker's Diagnostic Logs

N/A

[andreineculau]

If I may suggest a possible solution:

• set HOME only if not already set
• set GITHUB_HOME or GITHUB_RUNNER_HOME in order to allow devs not to hardcode /github/home in case there's a use-case to access the runner's home from inside a container.

[robin-rpr]

As we are also having trouble with this issue in our Build Actions, we used this workaround:

Note: We are expecting the /github/home folder to not exist yet in the container.

1. Create the /github directory
2. Create a Symlink with $HOME

See:

mkdir -p /github
ln -s /root $HOME # Create Symlink /github/home

[hoxu]

I created a bit similar issue about issues caused by setting HOME: #1146.

[andreineculau]

@Hoxu you touched on the most important bit:

On a related note, it would be really nice if the GitHub runner's parameters to docker containers were documented clearly somewhere.

[JJ]

Overriding $HOME causes all kinds of trouble, to no obvious benefit...

[vamper424]

Hola gracias salientes desde Los Mochis Sinaloa México. Soy mexicano y de familia numerosa saludos again asta luego.bestamosven línea.

[Taknok]

Also, at build the HOME is defined by image/distribution to /root for example. Then when running, the HOME is overwritten by GitHub to /home/github. This is causing issues with some tools installed in image.

[andreineculau]

I can update this thread for future readers with my current workaround, that doesn't require messing with the filesystem as in https://github.com/actions/runner/issues/863#issuecomment-757230259 , but does require sudo, maybe tee.

Add a first step to the job like

- shell: bash
  run: 'echo HOME=/home/sf | sudo tee -a $GITHUB_ENV'

FWIW the sudo tee is needed because $GITHUB_ENV will be owned by another user. If you want to skip tee, you can run sudo sh -c "echo HOME=/home/sf >> $GITHUB_ENV"

Following steps will start with the expected HOME, as well as source resource files correctly, like ~.profile ~/.bash_profile etc

[andreineculau]

On another topic, driven by https://github.com/actions/runner/issues/863#issuecomment-997443323 "no obvious benefit", I have started thinking what can such a benefit be and I've come with these possible benefits:

1. persist state between container startups/shutdowns
2. persist state with host
3. persist state between containers
4. control the environment via .profile, .bash_profile, .curlrc, etc

Let's start from the bottom, with 4. /github/home is empty. So the only way to reason around that is if Github wants NO environment, in case they found cases where settings in resource files create unexpected behaviour. Sounds far fetched.

Regarding 3 - there's a specific volumes setting. "You can use volumes to share data between services or other steps in a job." Interesting formulation. First of all, did anyone see if Github overrides $HOME for services as well? I didn't. Secondly "other steps in a job" ?! I have just ran a test, and Github doesn't shuts down the container after each step. State is maintained between steps, and furthermore, action steps (i.e. steps with uses:some/action@ref) also run in the same container, having access to the same state.

Regarding 2. Which host? Either you run the job in a container or in a real/virtual machine.

Regarding 1. As mentioned for 3., state seems to be maintained. There's one particular case that I haven't tested, because I don't know where to look to understand how it works, how it should be invoked - docker hub action https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-using-a-docker-hub-action . Like ok, you reference an image hosted by hub.docker.com, what then? The docs reference an alpine image. How do you run shell commands? Or do they mean that you can package a docker image with a action.yml file in the WORKDIR, and they would run that. I really don't know how that works. But it could be that they start another container based on that image, set HOME to the same /github/home mount, same with GITHUB_WORKSPACE and maybe others, and thus share state though your job and the action step run isolated otherwise.

Summary:

• ❓ couldn't find a benefit
• 🙏 can someone enlighten me if a service also has $HOME mounted as /github/home ?
• 🙏 can someone enlighten me how a docker hub action step is intended to be used, how it works?

** 🚨 THE BEST thing would be if someone at Github can spend some minutes just to clarify WHY? I'm starting from the premise that there is a benefit. Something that evades some of us. If we understand why, then maybe we can even start using Github in different/better ways.**

[andreineculau]

ping @TingluoHuang

[andreineculau]

after some googling, I found https://stackoverflow.com/a/56685281/465684

GitHub Actions support confirmed that all of /github persists throughout a workflow run.

As the comment says, there's no citation, no documentation around that, AND it also doesn't hold in reality. I ran a test here https://github.com/ysoftwareab/yplatform-test/actions/runs/1783086868 . test3 jobs depends on test1 and test2, and test3 doesn't have access to the test1 and test2 files, neither in the real $HOME, neither in /github/home, neither in $GITHUB_WORKSPACE. Afaik, one uses upload-artifacts and download-artifacts actions for sharing files between jobs.

Just wanted to be exhaustive.

[andreineculau]

Looking at the runner's sourcecode, there's another mention of this mount https://github.com/actions/runner/blame/9a829995e02d2db64efb939dc2f283002595d4d9/src/Runner.Worker/Handlers/ContainerActionHandler.cs#L174

I ran out of time so I cannot test this to be sure, but I guess the whole idea of overriding $HOME is to share state with a "docker container action" https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action 😞

[nikola-jokic]

Hi @andreineculau,

I will mark this as an enhancement since this is working as expected, but obviously, this change could add some value. I am putting it in the enhancement backlog so it is considered in the future enhancements 😊

[andreineculau]

@nikola-jokic why do you say it is working as expected? Where is it documented that "variables CANNOT be overridden" is normal behaviour?

As far as I, and others on this thread, can see - this is a BUG . A step can set new, but cannot override previous variables. That's some peculiar "expectation".

---

EDIT: ignore this comment, I thought I'm replying in the https://github.com/actions/runner/issues/789 thread

[nikola-jokic]

Hi @andreineculau,

We are experiencing similar problems with the Issue 434, which is related to how we mount the fs to have those files re-used. After the discussion with the team, we marked it as a feature that we should work on in the near future. That problem is also very important to us. This issue is related to the HOME environment, which is used to share the state between the host system where the runner is executing, and the containerized application.

Since issue 434 is considered a feature, I don't see why this issue would not fall under the same umbrella and should be picked up together.

[andreineculau]

@nikola-jokic Apologies. My comment above was wrong, because I thought we were talking about https://github.com/actions/runner/issues/789 (not an excuse but I have a cold and I wasn't thinking right).

I will read up on #434 and come back if needed. Thanks and once again sorry for the noise.

[nikola-jokic]

@andreineculau, no problem 😊. I misunderstood the issue 789, so I re-opened it once I got it. There is a third party PR now related to that issue so it should be fixed as soon as it is merged.

[klepiz]

Im running a script on the CI in order to set new environment variables (such as JAVA_HOME) but its seems like it doesn't work

- run: sh /usr/local/sbin/entrypoint.sh 

is that part of this issue? or is there a way to set env variable by a bash script?

Note: seems like the container is not overridden the variable if it was set as ENV statement in the dockerfiles, so that could be a possible workaround of this issue

[vitalytarasov]

Are there any updates on the issue? It has been almost a year without resolution.

[arg0d]

This is an awful bug in Github actions. Who thought its OK to overwrite $HOME variable? Overwriting $HOME variable breaks a lot of intuitive assumptions about Linux environment. What is the reason for overwriting $HOME variable? If overwriting $HOME variable does not benefit the users directly, this needs to be fixed ASAP.

[danielefranceschi]

This is a custom JS-only checkout action we use that contains two workarounds for self-hosted runners with containerized jobs:

• workspace cleanup at the end of the job (to avoid the file ownership mess, thanks to FraBle/clean-after-action@v1)
• $HOME overriding when the job is executed inside a container

name: 'custom-checkout'
description: 'source code checkout'

inputs:
  fetch-depth:
    description: 'Depth for git fetch'
    required: false
    default: 1
  lfs:
    description: 'Whether to download Git-LFS files'
    required: false
    default: false

outputs:
  homedir:
    description: 'home directory'
    value: ${{ steps.setup-homedir.outputs.result }}

runs:
  using: "composite"
  steps:
    - name: setup homedir
      id: setup-homedir
      uses: actions/github-script@v6
      env:
        CONTAINER_ID: ${{ job.container.id }}
      with:
        result-encoding: string
        script: |
          const z = process.env.CONTAINER_ID;
          if (z) {
            const newhome = `/tmp/home`;
            await io.mkdirP(newhome);
            core.exportVariable("HOME", newhome);
            return newhome;
          }
          return process.env.HOME;
    - uses: FraBle/clean-after-action@v1
    - uses: actions/checkout@v3
      with:
        fetch-depth: ${{ inputs.fetch-depth }}
        lfs: ${{ inputs.lfs }}

To be used everywhere a runs-on: [self-hosted,linux] is present.

[areusch]

utmost confusion, especially when other JS-based GitHub actions as well as some subcommands continue to think the container's homedir is what it says in /etc/passwd (i.e. /root or whatever your container sets), but then running stuff in shells populates /github/home.

also, figuring this out via a 5 minute edit-refresh loop was also utmost joy.

is this...necessary...?

[gabyx]

Can this be fixed in 2024, this should not be any feature or something, this is just totally non-sense and is a bug! The container defines the HOME variable, full-stop.

[rlindner81]

After reading this whole thread, I'm really disappointed that this got re-labeled as an enhancement. This behavior is not only unintuitive and pointlessly disruptive to most non-trivial docker images, it's also something that thousands of Github Action users will run into and be completely confused by. That's real suffering.

[kdvolder]

Here's a concrete example of problems caused by the 'unexpected' value of $HOME.

Code like this doesn't do what you would expect:

ssh-keyscan -t rsa some.example.domain >> ~/.ssh/known_hosts

You'd expect this will avoid 'unknown host key' for ssh to the host in question. Sadly it doesn't and the reason appears to be that ~/.ssh resolves to /github/home/.ssh but ssh itself is looking instead (in our case) in /home/ghrunner/.ssh (which is where the actual home directory of the 'ghrunner' user is).

[kyledecot]

@kdvolder Exactly my experience. Anywhere where tilde is used you need to "expand" it manually to be the "real" home, not the hijacked one.

My Dockerfile is full of things like:

ENV RUSTUP_HOME=/workflow/.rustup \
    NPM_CONFIG_PREFIX=/workflow/.npm \
    DOCKER_CONFIG=/workflow/.docker \
    CARGO_HOME=/workflow/.cargo
…

This is a sub-optimal “solution” at best. In some cases the tool internally uses tilde and there is no way to “re-hijack” is back to the “real” HOME.

[AlexSkrypnyk]

I've added this as a very first step for each of my jobs in the workflow:

    steps:
      - name: Preserve $HOME set in the container
        run: echo HOME=/root >> "$GITHUB_ENV"

/root is a $HOME in my container.

This has to be the very first step in the job and it should not be a symlink as suggested above or the contents of your $HOME in the container will be overwritten by steps in the actions that precede this step.

[horror-proton]

# these gives /root
getent passwd | grep $USER
echo ~root

# while these gives /github/home
echo ~
echo $HOME

That's hilarious :disappointed:. Not all applications use environment variables to determine home directory (e.g. openssh). One must be very patient to find out what's going on with them.

[Yurii-huang]

I don't think it's appropriate to redirect to the root directory, some builds can specify user in options, for example, archlinux does not allow root to build, for the default home directory should determine whether there is a --user designation in options as a judgment condition

[Yurii-huang]

@andreineculau