Self-hosted runner environment variables are not available during docker container setup

[Sebastian-0]

Describe the bug If you define environment variables in a .env file for a self-hosted runner they can be accessed as $VARIABLE during job steps, but if you want to access the variable when setting up the job in a docker container it is not resolved and instead remains as the string $VARIABLE rather than the value of VARIABLE. See the following example.

For a self hosted runner .env file such as:

DOCKER_GPUS=device=0

And a job defined as

jobs:
  run_test:
    name: Run
    runs-on: self-hosted
    container:
      image: ubuntu
      options: --gpus $DOCKER_GPUS
    steps:
      - name: Print Hello World
        run: echo "Hello World!"

Running this will fail because $DOCKER_GPUS will be passed as a string to the docker --gpus flag, see excerpt from the job log:

Why is this a problem? We have servers with multiple GPUs and one self-hosted runner per GPU. Since self-hosted runners don't support docker-in-docker they all run on bare metal, while all the jobs are launched in docker containers using a setup like the above. We don't want to give all jobs access to all gpus (--gpus all) for multiple reasons, so therefore we defined the .env variables for each runner to be device=0, device=1, etc... to only expose a single GPU to jobs running on each runner. However, due to the problem above this doesn't work.

Further, this issue also means that you can't pass the runners' env variables into the docker container if you need to, like --env "MAX_CORES=$MAX_CORES", and similarly doing as follows also doesn't work:

container:
  image: ubuntu
  env:
    cores: $MAX_CORES

Of course running a job without a container gives access to the env variable as expected:

jobs:
  run_test:
    name: Run
    runs-on: self-hosted
    steps:
      - name: Print GPUs
        run: echo "$DOCKER_GPUS"

To Reproduce Steps to reproduce the behavior:

1. Use a machine with a GPU installed
2. Create a self hosted runner and add DOCKER_GPUS=device=0 to the .env file
3. Create a workflow containing my first example above
4. Run the workflow and see that it fails to read the environment variable

Alternatively if you don't have a GPU available you can also try to pass in an environment variable instead using --env "A=$B" and print its contents in the step, this will also fail.

Expected behavior The variables in the .env file of the self-hosted runner should be accessible when the docker container is launched. If so, my example workflow above would have completed successfully.

Runner Version and Platform

Runner version: 2.313.0 Hosted on Ubuntu 22.04 machines.

What's not working?

See above.

Job Log Output

Log output on failure (same image as above):

Runner and Worker's Diagnostic Logs

Not relevant, can complement if necessary.

[robsoncombr]

+1

using docker container, docker.n8n.io/n8nio/n8n:1.36.2, vars requires enterprise plan

[yanchenmochen]

Thanks very much, I am also confronted with the same problem, How to separate GPUs for different runner process, any way, Thankyou for giving the insight about how to CI in a single server for multiple github repositories

[halradaideh]

same here trying to pick up a token at runner runtime

jobs:
  container-based:
    name: test
    runs-on: [test-runners]
    container:
      image: X
      credentials:
       username: oauth2accesstoken
       password: ${{ token }}

even when I export it before the /home/runner/run.sh script