`RUNNER_TOOL_CACHE` and/or `AGENT_TOOLSDIRECTORY` not picked up on self hosted runner

[nvincent-vossloh]

Description: I have exported RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY in the shell running the self-hosted runner, however a pip install in the workflow still uses /opt/hostedtoolcache directory. job-log.txt

2024-07-23T09:20:33.0492190Z       [1/41] Compiling C object gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0493604Z       FAILED: gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0502062Z       cc -Igi/_gi.cpython-310-x86_64-linux-gnu.so.p -Igi -I../gi -I. -I.. -I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/
usr/include/gobject-introspection-1.0 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -DPY_SSIZE_T_CLEAN -Wall -Warray-bounds -Wcast-align -Wduplicated-branches -Wextra -Wform
at=2 -Wformat-nonliteral -Wformat-security -Wimplicit-function-declaration -Winit-self -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wmissing-prototypes -Wne
sted-externs -Wnull-dereference -Wold-style-definition -Wpacked -Wpointer-arith -Wrestrict -Wreturn-type -Wshadow -Wsign-compare -Wstrict-aliasing -Wstrict-prototypes -Wswitch-default -Wundef -Wunused-but-set-variable -Wwrite-strings -W
no-missing-field-initializers -Wno-unused-parameter -Wno-discarded-qualifiers -Wno-sign-conversion -Wno-cast-function-type -Wno-int-conversion -fno-strict-aliasing -fvisibility=hidden -MD -MQ gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -MF gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o.d -o gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -c ../gi/pygboxed.c
2024-07-23T09:20:33.0511885Z       cc1: warning: /opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10: No such file or directory [-Wmissing-include-dirs]
2024-07-23T09:20:33.0513997Z       ../gi/pygboxed.c:23:10: fatal error: Python.h: No such file or directory
2024-07-23T09:20:33.0515317Z          23 | #include <Python.h>
2024-07-23T09:20:33.0516190Z             |          ^~~~~~~~~~
2024-07-23T09:20:33.0517073Z       compilation terminated.

Action version: actions/setup-python@v5

Platform:

• [x] Ubuntu
• [ ] macOS
• [ ] Windows

Runner type:

• [ ] Hosted
• [x] Self-hosted

Tools version: python-version: '3.10'

Repro steps:

name: setup python on shr    
# Test pip install which requires compilation on host where include path is not correct    
# https://github.com/actions/setup-python/issues/485    
# https://github.com/actions/setup-python/issues/489    
    
on:    
  # Allows you to run this workflow manually from the Actions tab    
  workflow_dispatch:    
  push:    
    branches:    
      - setup-python-pip-install    
    
jobs:    
  test-python:    
    name: Test Python pip install    
    runs-on: [self-hosted, linux, test-python]    
    steps:    
      - name: Github Actions Context Information    
        uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7    
              
      - uses: actions/setup-python@v5    
        with:    
          python-version: '3.10'    
    
      - name: show env    
        run: |    
          env    
    
      - name: python pkg install    
        run: |    
          pip install PyGObject

From the Self-hosted runner directory:

export RUNNER_TOOL_CACHE="$(pwd)/_work/_tool"
export AGENT_TOOLSDIRECTORY="$(pwd)/_work/_tool"
./run.sh

Expected behavior: I would expect the pip install step to set properly the -I flags without /opt/hostedtoolcache

Actual behavior: The logs (see above) show that it tried to add an include path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which do not exist on the self hosted runner.

Workaround I am not fond of

export CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10
./run.sh

With this workaround I manage to install the lib, however it looks like that pip cached some stuff in the default cache directory:

  pip install PyGObject
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    PKG_CONFIG_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib/pkgconfig
    Python_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    Python2_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    Python3_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    LD_LIBRARY_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib
Collecting PyGObject
  Using cached pygobject-3.48.2.tar.gz (715 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pycairo>=1.16 (from PyGObject)
  Using cached pycairo-1.26.1-cp310-cp310-linux_x86_64.whl
Building wheels for collected packages: PyGObject
  Building wheel for PyGObject (pyproject.toml): started
  Building wheel for PyGObject (pyproject.toml): finished with status 'done'
  Created wheel for PyGObject: filename=pygobject-3.48.2-cp310-cp310-linux_x86_64.whl size=259354 sha256=fc1421f9adae4077d60198ee04309773969c33b2372d1cc93f2733967c3f3f96
  Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97
Successfully built PyGObject
Installing collected packages: pycairo, PyGObject
Successfully installed PyGObject-3.48.2 pycairo-1.26.1

(Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97) even though the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY have been set up.

I took the idea of this workaround from https://github.com/actions/setup-python/issues/489 and https://github.com/actions/setup-python/issues/485 However I do not understand why the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY are not used.

Beside, if i run

grep -R /opt/hostedtoolcache *

from _work/_tool I get a lot of match:

...
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIGURE_LDFLAGS=      -Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:prefix=         /opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIG_ARGS=     '--prefix=/opt/hostedtoolcache/Python/3.10.14/x64' '--enable-shared' '--enable-optimizations' '--enable-loadable-sqlite-extensions' 'LDFLAGS=-Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib'
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/python-config.py:#!/opt/hostedtoolcache/Python/3.10.14/x64/bin/python3.10
grep: Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.a: binary file matches
Python/3.10.14/x64/lib/pkgconfig/python-3.10-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python-3.10.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
...

I don't really know where pip gets the include path from, but I guess from one of the above files which clearly points to an invalid folder.

Thanks in advance for your help, let me know if you need more information.