Non Portable binary, doesn't work with sudo or without LD_LIBRARY_PATH

[jaswanthikolla]

Description: Python binary is compiled with rpath that's /opt/hostedtoolcache/Python. Now, latest github runners define RUNNER_TOOL_CACHE/AGENT_TOOLSDIRECTORY differently than /opt/hostedtoolcache and that installs python at /home/runner/_work/_tool/Python/. So, with this, there are 2 issues.

1. sudo python --version doesn't work ( See Error section) because most systems's doesn't allow passing LD_LIBRARY_PATH due to security issues.
2. python --version doesn't work without setting environment variable LD_LIBRARY_PATH

output of ldd :

runner@arss-runner-xxxx-runner-5jvhz:~/_work/_tool/Python/3.12.3/x64/bin$ ldd python3
	linux-vdso.so.1 (0x00007ffceb776000)
	libpython3.12.so.1.0 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7ccc75c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7ccc991000)

Error:

python: error while loading shared libraries: libpython3.9.so.1.0: 
cannot open shared object file: No such file or directory

What could be the reason:

rpath is hardcoded to /opt/hostedtoolcache/Python. if we use $ORIGIN, binaries will be portable. Fix is raised here https://github.com/actions/python-versions/pull/275

Action version: v5.1.0

Platform:

• [x] Ubuntu
• [ ] macOS
• [ ] Windows

Runner type:

• [ ] Hosted
• [x] Self-hosted

Tools version: All versions of Python 3.9.x, 3.10.x, etc.

Repro steps:
You can easily reproduce using following steps. And also, you can unset LD_LIBRARY_PATH and just use without sudo

    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: |
          sudo python --version

Expected behavior: There are times we need to use sudo python and Most system ( also not acceptable to pass) doesn't pass LD_LIBRARY_PATH as environment variable due to security issues.

Actual behavior: sudo python --version to work.

[aparnajyothi-y]

Hello @jaswanthikolla, Thank you for creating this issue and we will get back to you once we have some feedback on this :)

[jaswanthikolla]

@aparnajyothi-y I raised the issue just for documentation. PR is already raised to fix this issue, Can you review that.

[navarro967]

@jaswanthikolla thank you for your efforts thus far, it's a shame there hasn't been more movement on this.

@aparnajyothi-y This is affecting certain ansible modules as well now. I am unable to run my checks now due to this.

[jaswanthikolla]

@priya-kinthali Any update on the PR as it's been 2 months? I see reduced activity with so many open PRs, also see that It's outsourced to some other company. Is this repo still maintained, if not better to handover to other companies which can manage it.

[priya-kinthali]

Hello @jaswanthikolla 👋, Thank you for providing a detailed description of your issue, and we apologize for the delayed response. For GitHub runners, the default path is /opt/hostedtoolcache, whereas for self-hosted runners, the runner downloads and installs tools into the folder set up by the RUNNER_TOOL_CACHE environment variable. On Linux self-hosted runners, you can change this location by setting the AGENT_TOOLSDIRECTORY environment variable. This allows you to specify any custom directory to install Python on your self-hosted runner using this variable. For more details, refer to the advanced usage documentation.

Here is a potential workaround that you might consider.

- name: Create Python temp_script.sh  and check python version
  run: |
    PYTHON_PATH=$(which python3)
    PYTHON_DIR=$(dirname $PYTHON_PATH)
    echo "export LD_LIBRARY_PATH=$PYTHON_DIR/../lib" >> temp_script.sh
    echo "export PATH=$PYTHON_DIR:\$PATH" >> temp_script.sh
    echo "python --version" >> temp_script.sh
    chmod +x temp_script.sh
    sudo ./temp_script.sh       

Thank you again for the PR. We have already started reviewing it, and testing is underway to check its impact on the overall build environment for both GitHub and self-hosted runners. We will provide feedback once we have tested all possible scenarios.

[maxim-lobanov]

Hi @jaswanthikolla , side question re

Now, latest github runners define RUNNER_TOOL_CACHE/AGENT_TOOLSDIRECTORY differently than /opt/hostedtoolcache and that installs python at /home/runner/_work/_tool/Python/

On which runners do you see such behaviour? I would expect all github hosted runners to have tool cache under /opt/hostedtoolcache

[priya-kinthali]

Hello @jaswanthi, Just a gentle ping to follow up on the previous conversations. Have you had a chance to try the suggested workaround?

On which runners do you see such behaviour? I would expect all github hosted runners to have tool cache under /opt/hostedtoolcache

Additionally could you please confirm? Thankyou!

[jaswanthikolla]

@maxim-lobanov For ARSS runners, the default path is /home/runner/_work/_tool/Python/ .

Yes, that workaround will work, but we pursued another workaround internally. Following up if there are any updates on the PR as It's been 3 months. I can't maintain context on it forever to address review comments and so I guess it's time to abandon/close the PR.

[klalumiere]

I'm not exactly sure why there is a lengthy discussion around this issue. Using ORIGIN is typical in the *nix world, and using hardcoded path (except maybe the standard one like /usr/local/lib) is usually seen as an anti-pattern. In the current case, using a hardcoded path prevents actions/setup-python from being used easily on self-hosted GitHub runners (our case).

Of course there are workarounds, but they decrease the quality of user experience and they increase the maintenance burden of users for their actions and workflows. So let's reverse the question: what are the downsides of starting to use ORIGIN as already implemented in the pull request when compared to the status quo? 🤔

If there is no significant downsides, why not just benefit from the community's work and merge the pull request ASAP? 😎 :shipit:

[navarro967]

@klalumiere I've resorted to forking both repositories and using that for now. It's been months without a resolution despite @jaswanthikolla's efforts. Her PR works and should be merged.

[WTPOptAxe]

Here is a potential workaround that you might consider.

- name: Create Python temp_script.sh  and check python version
  run: |
    PYTHON_PATH=$(which python3)
    PYTHON_DIR=$(dirname $PYTHON_PATH)
    echo "export LD_LIBRARY_PATH=$PYTHON_DIR/../lib" >> temp_script.sh
    echo "export PATH=$PYTHON_DIR:\$PATH" >> temp_script.sh
    echo "python --version" >> temp_script.sh
    chmod +x temp_script.sh
    sudo ./temp_script.sh       

Thank you again for the PR. We have already started reviewing it, and testing is underway to check its impact on the overall build environment for both GitHub and self-hosted runners. We will provide feedback once we have tested all possible scenarios.

This workaround does not work on self-hosted runners with pantsbuild.

[leofang]

We're also affected by this (with our self-hosted runners), did not realize by the end of 2024 there are still real-world Python compiled without relocatability 😢 This is very frustrating for people trying to rely more on GHA and less on custom Python distros (ex: conda)...

[zanieb]

Hi

I'd like to raise another problem with the LD_LIBRARY_PATH approach — this overrides RUNPATH which means that you can break other Python interpreters on the system. For example, I'm working on musl distributions of CPython over in https://github.com/astral-sh/python-build-standalone/pull/541 and our tests failed in CI because LD_LIBRARY_PATH is set and our Python interpreter loaded the GitHub Actions libpython instead of ours. A bit of an side, but we can't use DT_NEEDED (which takes precedence over LD_LIBRARY_PATH) instead of RUNPATH there because musl doesn't support the $ORIGIN syntax in it.

Using $ORIGIN really is best practice here. If there are downsides to that approach, I'd love to hear it. We've been using $ORIGIN for relocatable Python distributions in python-build-standalone for years now without issue.

[mike-essential]

@priya-kinthali please merge this