python: remove `LD_LIBRARY_PATH` hack from running python environment

[vlaci]

Injecting LD_LIBRARY_PATH to the Python runtime environment is great to bypass the need of having to patch non-nix binaries loaded into that environment, however it breaks down, when Python executes any other program not compiled for the given Nix system, e.g. a shell script via subprocess.

To work this around, devenv will inject a pth^1 file to the virtual environment it creates, which mangles the LD_LIBRARY_PATH variable, undoing any changes to it made by devenv but preserving changes from other sources.

I am unsure what the best way would be to integrate this, bit I think the approach in general is sane. I am tested only the poetry case currently.

Fixes #1111

[bobvanderlinden]

I was thinking, since this will probably be needed just for Linux, that it might be nice to eventually have an LD_PRELOAD that removed the path from LD_LIBRARY_PATH. That way this exact same trick can be applied to ruby, nodejs and more.

Still useful to have this pth to see how it fares in the python world.

[domenkozar]

Can we get some tests for this?

[vlaci]

I've been testing this change in a few projects, and I have mixed results with it:

• in general everything is fine, when running from the venv
• there are some edge cases, if some tool creates a new virtualenv, it will be completely broken (like pre-commit when no nix integration is used)

I am also testing a completely different approach, by using patchelf instead: https://github.com/cachix/devenv/compare/main...vlaci:devenv:python-venv-patchelf

It has its issues, like pip install in the venv won't recheck for binaries to patch, otherwise it is also a viable approach.

It may also be implemented as an add-on, so python packages providing binaries, like ruff also work.

[dsalaza4]

Hi,

I think this PR relates to https://github.com/cachix/devenv/issues/1699.

I was not aware that languages.python was injecting LD_LIBRARY_PATH to the runtime.

What I noticed is that when using languages.python.poetry.enable = true and providing a pyproject.toml and poetry.lock with ruff = "0.8.0", the fetched ruff binary from pypi does not work properly on the devenv container as it looks for linking libraries in /lib and /lib64.

[vlaci]

Hi,

I think this PR relates to #1699.

I was not aware that languages.python was injecting LD_LIBRARY_PATH to the runtime.

What I noticed is that when using languages.python.poetry.enable = true and providing a pyproject.toml and poetry.lock with ruff = "0.8.0", the fetched ruff binary from pypi does not work properly on the devenv container as it looks for linking libraries in /lib and /lib64.

I've extracted out my auto-patchelf PoC[^1], and it works fine with current devenv versions:

https://github.com/vlaci/devenv-extras

[^1]: PoC: https://github.com/cachix/devenv/compare/main...vlaci:devenv:python-venv-patchelf

[bobvanderlinden]

@vlaci I think that solution is similar to https://github.com/cachix/devenv/pull/840. LD_LIBARY_PATH was preferred over autopatchelf at the time. I do still regularly run into situations where autopatchelf would be a great option. It's why I proposed https://github.com/cachix/devenv/pull/1542, so that the autopatchelf-solution can be built on top of devenv (allowing removal of the LD_LIBRARY_PATH python wrapper).

[leiserfg]

Sometimes LD_LIBARY_PATH is the only way, like when using python packages that load the lib with ctypes or cffi.

[domenkozar]

Sometimes LD_LIBARY_PATH is the only way, like when using python packages that load the lib with ctypes or cffi.

Hence I proposed a long time ago https://discuss.python.org/t/pkgconfig-specification-as-an-alternative-to-ctypes-util-find-library/31379