actions
`RUNNER_TOOL_CACHE` and/or `AGENT_TOOLSDIRECTORY` not picked up on self hosted runner
Description:
I have exported RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY in the shell running the self-hosted runner, however a pip install in the workflow still uses /opt/hostedtoolcache directory.
job-log.txt
2024-07-23T09:20:33.0492190Z [1/41] Compiling C object gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0493604Z FAILED: gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0502062Z cc -Igi/_gi.cpython-310-x86_64-linux-gnu.so.p -Igi -I../gi -I. -I.. -I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/
usr/include/gobject-introspection-1.0 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -DPY_SSIZE_T_CLEAN -Wall -Warray-bounds -Wcast-align -Wduplicated-branches -Wextra -Wform
at=2 -Wformat-nonliteral -Wformat-security -Wimplicit-function-declaration -Winit-self -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wmissing-prototypes -Wne
sted-externs -Wnull-dereference -Wold-style-definition -Wpacked -Wpointer-arith -Wrestrict -Wreturn-type -Wshadow -Wsign-compare -Wstrict-aliasing -Wstrict-prototypes -Wswitch-default -Wundef -Wunused-but-set-variable -Wwrite-strings -W
no-missing-field-initializers -Wno-unused-parameter -Wno-discarded-qualifiers -Wno-sign-conversion -Wno-cast-function-type -Wno-int-conversion -fno-strict-aliasing -fvisibility=hidden -MD -MQ gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -MF gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o.d -o gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -c ../gi/pygboxed.c
2024-07-23T09:20:33.0511885Z cc1: warning: /opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10: No such file or directory [-Wmissing-include-dirs]
2024-07-23T09:20:33.0513997Z ../gi/pygboxed.c:23:10: fatal error: Python.h: No such file or directory
2024-07-23T09:20:33.0515317Z 23 | #include <Python.h>
2024-07-23T09:20:33.0516190Z | ^~~~~~~~~~
2024-07-23T09:20:33.0517073Z compilation terminated.
Action version:
actions/setup-python@v5
Platform:
- [x] Ubuntu
- [ ] macOS
- [ ] Windows
Runner type:
- [ ] Hosted
- [x] Self-hosted
Tools version: python-version: '3.10'
Repro steps:
name: setup python on shr
# Test pip install which requires compilation on host where include path is not correct
# https://github.com/actions/setup-python/issues/485
# https://github.com/actions/setup-python/issues/489
on:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
push:
branches:
- setup-python-pip-install
jobs:
test-python:
name: Test Python pip install
runs-on: [self-hosted, linux, test-python]
steps:
- name: Github Actions Context Information
uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: show env
run: |
env
- name: python pkg install
run: |
pip install PyGObject
From the Self-hosted runner directory:
export RUNNER_TOOL_CACHE="$(pwd)/_work/_tool"
export AGENT_TOOLSDIRECTORY="$(pwd)/_work/_tool"
./run.sh
Expected behavior:
I would expect the pip install step to set properly the -I flags without /opt/hostedtoolcache
Actual behavior:
The logs (see above) show that it tried to add an include path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which do not exist on the self hosted runner.
Workaround I am not fond of
export CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10
./run.sh
With this workaround I manage to install the lib, however it looks like that pip cached some stuff in the default cache directory:
pip install PyGObject
shell: /usr/bin/bash -e {0}
env:
pythonLocation: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
PKG_CONFIG_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib/pkgconfig
Python_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
Python2_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
Python3_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
LD_LIBRARY_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib
Collecting PyGObject
Using cached pygobject-3.48.2.tar.gz (715 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pycairo>=1.16 (from PyGObject)
Using cached pycairo-1.26.1-cp310-cp310-linux_x86_64.whl
Building wheels for collected packages: PyGObject
Building wheel for PyGObject (pyproject.toml): started
Building wheel for PyGObject (pyproject.toml): finished with status 'done'
Created wheel for PyGObject: filename=pygobject-3.48.2-cp310-cp310-linux_x86_64.whl size=259354 sha256=fc1421f9adae4077d60198ee04309773969c33b2372d1cc93f2733967c3f3f96
Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97
Successfully built PyGObject
Installing collected packages: pycairo, PyGObject
Successfully installed PyGObject-3.48.2 pycairo-1.26.1
(Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97)
even though the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY have been set up.
I took the idea of this workaround from https://github.com/actions/setup-python/issues/489 and https://github.com/actions/setup-python/issues/485
However I do not understand why the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY are not used.
Beside, if i run
grep -R /opt/hostedtoolcache *
from _work/_tool I get a lot of match:
...
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIGURE_LDFLAGS= -Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:prefix= /opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIG_ARGS= '--prefix=/opt/hostedtoolcache/Python/3.10.14/x64' '--enable-shared' '--enable-optimizations' '--enable-loadable-sqlite-extensions' 'LDFLAGS=-Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib'
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/python-config.py:#!/opt/hostedtoolcache/Python/3.10.14/x64/bin/python3.10
grep: Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.a: binary file matches
Python/3.10.14/x64/lib/pkgconfig/python-3.10-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python-3.10.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
...
I don't really know where pip gets the include path from, but I guess from one of the above files which clearly points to an invalid folder.
Thanks in advance for your help, let me know if you need more information.
Hello @nvincent-vossloh, Thank you for creating this issue. We will investigate it and provide feedback as soon as we have some updates.
Hello @nvincent-vossloh, Thank you once again for creating this issue. We have investigated and found that the problem is due to environment variables not being available in the context where the action is running. The given logs indicate the inclusion of a path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which does not exist on the self-hosted runner. You need to ensure that the actions/setup-python action respects the custom tool cache directory by setting the environment variables directly in the workflow, as shown in the job below:
jobs:
test-python:
name: Test Python pip install
runs-on: [self-hosted, linux, test-python]
env:
RUNNER_TOOL_CACHE: ${{ github.workspace }}/_work/_tool
AGENT_TOOLSDIRECTORY: ${{ github.workspace }}/_work/_tool
steps:
- name: Github Actions Context Information
uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Show environment variables
run: env
- name: Python package install
run: pip install PyGObject
you can use any custom directory to install python on your self-hosted runner using AGENT_TOOLSDIRECTORY environment variable as per the setup-python documentation. Pip gets the include path from the environment variables set up during the installation of Python. Specifically, the paths such as PKG_CONFIG_PATH, Python_ROOT_DIR, Python2_ROOT_DIR, and Python3_ROOT_DIR are configured to point to the respective directories during the setup process.
Hello @nvincent-vossloh, Please let us know in case of any further concerns on the above information.
Thanks for your feedback, I will try your solution and let you know if that fixed the issue on my side.
The documentation you link to, states the the environment variable could be set in the same shell where the runner is running. Unfortunately that does not work (anymore ?), that is why I opened the issue in the first place. Maybe the runner software does not use the env variable from the shell anymore (did not work with the .env file either).
Anyway, unfortunately setting the env variable in the workflow does not fix the issue, the behavior is still the same. On my first try it succeeded, but that was because the pip install step used a cache version of PyGObject, after deleting it the failure to compile still happens.
Only setting CPATH seems to work.
Hi @nvincent-vossloh, After further investigation, we found that setting the environment variable in the workflow successfully works when the cached version of PyGObject is available. However, after clearing the cache, the compilation failure reoccurs. Given these circumstances, It appears that only setting CPATH directly works. This is because CPATH explicitly sets the include path for the compiler, ensuring the correct headers are found during the compilation of PyGObject. The default paths set by pip or setup-python may not align with the directory structure on the self-hosted runner, leading to the observed compilation errors.
Setting CPATH ensures that the compiler can locate the necessary Python header files, avoiding the "No such file or directory" error for Python.h. This approach provides a direct and reliable way to specify include directories, bypassing any inconsistencies in environment variable handling between the workflow and the self-hosted runner.
Thank you for your understanding and patience as we work through this issue.
Do you plan on updating the setup-python action in order to add the include directory to the CPATH env var ?
Hi @nvincent-vossloh, Thank you for suggesting to include directory to the CPATH environment variable in the setup-python action. We acknowledge the importance of this feature and will explore the possible ways to implement it. We will consider this for a future enhancement.
In the meantime, here is an alternative solution to include CPATH in a GitHub Actions workflow: Modify the workflow to set the CPATH environment variable: The CPATH environment variable can be set directly in the workflow file using the env key.
jobs:
test-python:
name: Test Python pip install
runs-on: self-hosted
steps:
- name: Github Actions Context Information
uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Set CPATH environment variable
run: echo "CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10" >> $GITHUB_ENV
- name: Python package install
run: pip install PyGObject
Hi @nvincent-vossloh, Thank you for suggesting to include directory to the CPATH environment variable in the setup-python action. We acknowledge the importance of this feature and will explore the possible ways to implement it. We will consider this for a future enhancement.
Thank you for taking my suggestion into account.
In the meantime, here is an alternative solution to include CPATH in a GitHub Actions workflow: Modify the workflow to set the CPATH environment variable: The CPATH environment variable can be set directly in the workflow file using the env key.
jobs: test-python: name: Test Python pip install runs-on: self-hosted steps: - name: Github Actions Context Information uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7 - uses: actions/setup-python@v5 with: python-version: '3.10' - name: Set CPATH environment variable run: echo "CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10" >> $GITHUB_ENV - name: Python package install run: pip install PyGObject
In the end I used this:
env:
pytest_python_version: '3.10'
[...]
- uses: actions/setup-python@v5
id: setup-python
with:
python-version: ${{env.pytest_python_version}}
- name: setup CPATH
run: |
echo "python version : ${{ steps.setup-python.outputs.python-version }}"
python_version="${{ env.pytest_python_version }}"
echo "python path : ${{ steps.setup-python.outputs.python-path }}"
CPATH=${pythonLocation}/include/python${python_version}
echo CPATH=${CPATH} >> "${GITHUB_ENV}"
which leverage the various env variables set.
Thanks again.