`PYO3_PYTHON` and `LD_LIBRARY_PATH` pointing to `venv` are ignored

[jakob-lilliemarck]

Bug Description

I would like pyo3 to use the .venv I've created in my python library, when calling functions from that library from rust. - but it ignores the values configured in my environment.

From what I've read I understood it as PYO3_PYTHON and/or LD_LIBRARY_PATH can be used to configure the path to the python interpreter executable. I exported them in the shell and then ran my rust project like so

export PYO3_PYTHON=/path/to/my/python/project/.venv/bin/python
export LD_LIBRARY_PATH=/path/to/my/python/project/.venv/bin/python
cargo run --bin fxctl ping

ping is just a function I use to test the setup. In the rust app, the ping function is invoked through a clap command. The handler looks like so:

use pyo3::prelude::*;

pub fn ping() -> PyResult<()> {
    Python::with_gil(|py| {
        let sys = PyModule::import(py, "sys").unwrap();

        let py_path: String = sys.getattr("executable").unwrap().extract().unwrap();
        let py_version: String = sys.getattr("version").unwrap().extract().unwrap();

        println!("path: {}\nversion: {}", py_path, py_version);
    });
    Ok(())
}

Steps to Reproduce

Reproduction repository: https://github.com/jakob-lilliemarck/pyo3-python-env-ignored

Backtrace

There is no backtrace as there is no actual error

Your operating system and version

Linux 6.6.65-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Dec 11 22:24:04 UTC 2024 x86_64 GNU/Linux

Your Python version (python --version)

3.12.7 (main, Oct 1 2024, 11:15:50) [GCC 14.2.1 20240910]

Your Rust version (rustc --version)

rustc 1.85.0-nightly (a224f3807 2024-12-09)

Your PyO3 version

0.23.3

How did you install python? Did you use a virtualenv?

Can't recall, but from the looks of it I installed the global one using brew, then I created a virtual env in my project by python -m venv .venv

Additional Info

No response

[Hennzau]

Hi,
I’m not sure if it’s a typo, but the export of PYO3_PYTHON should use one =:

export PYO3_PYTHON=/path/to/my/python/project/.venv/bin/python

Regarding LD_LIBRARY_PATH, it should point to the directory containing libpython.so. This is typically not found in the .venv but rather in the global installation, for example:

/home/linuxbrew/.linuxbrew/opt/[email protected]/lib

My workaround is to run cargo build/run after activating my venv and i add the ld path of my global python (see #4813)

[jakob-lilliemarck]

@Hennzau thanks for your reply. That's indeed a typo, I edited it to avoid confusion. However the issue persists, I've set up a simple reproduction repository. See the README for steps to install and run it. https://github.com/jakob-lilliemarck/pyo3-python-env-ignored

Exporting PYO3_PYTHON in the terminal before running with cargo seems to make no difference. However, if I activate the venv prior to running the rust program then the path is correct. . .venv/bin/python.

I also tried to call a shell in the main.rs to see if I could activate the virtualenv from Rust, while I got exitcode 0, that did not change the executable path that PyO3 discovered.

My expectation would be that I could point PyO3 to a virtualenv and that it should then activate and use that virtualenv.

[Hennzau]

Hi, the easiest solution is to always activate your venv before running cargo build/run.

Once pyo3 is compiled you cannot change anything from your main.rs, the only thing you can try is doing all your things in a build.rs script.

If you don't want to activate your venv:

I tried something with the PYO3_CONFIG_FILE env variable, it compiles successfully but I can't run it (it fails). The idea is to create a config file:

implementation=CPython
version=3.13
shared=true
abi3=false
lib_name=python3.13
lib_dir=/home/USER/path/to/lib/cpython-3.13.1-linux-x86_64-gnu/lib
executable=/home/USER/path/to/pyo3-python-env-ignored/.venv/bin/python
pointer_width=64
build_flags=
suppress_build_script_link_lines=false

and pass it while running cargo build:

PYO3_CONFIG_FILE=$(pwd)/pyo3_config_file cargo build

It should build successfully but then when i run it, I get:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ec07a612180 (most recent call first):
  <no Python frame>

[jakob-lilliemarck]

Once pyo3 is compiled you cannot change anything from your main.rs, the only thing you can try is doing all your things in a build.rs script.

hmm :thinking: I was under the impression that the python binary was linked at runtime, not during compilation..? Clearly I don't really understand how it works. The docs state the following:

PyO3 uses a build script (backed by the pyo3-build-config crate) to determine the Python version and set the correct linker arguments. By default it will attempt to use the following in order:

Any active Python virtualenv.
The python executable (if it's a Python 3 interpreter).
The python3 executable.

You can override the Python interpreter by setting the PYO3_PYTHON environment variable, e.g. PYO3_PYTHON=python3.7, PYO3_PYTHON=/usr/bin/python3.9, or even a PyPy interpreter PYO3_PYTHON=pypy3.

reference: https://pyo3.rs/main/building-and-distribution

Reading that it seems that configuring PYO3_PYTHON should indeed take precedence over the default order. However it is not really clear to me if PYO3_PYTHON variable is intended to be used during buildtime or runtime.

The use case I have is that I want to write a rust server application (with tokio) and then have that server application invoke functions in a python library. Having to activate the venv before starting the rust server doesn't seem ideal.

While calling a python library from Rust may seem overly complicated, I was hoping to be able to do some machine learning in my python lib (using np and keras) while keeping everything database- and infrastructure-related on the Rust side.

[jakob-lilliemarck]

Alright, turns out this isn't a bug at all :face_with_peeking_eye: please feel free to close this ticket.

PYO3 seems to do exactly what can be expected of it, the issue was due to my lack of understanding of the python interpreter and virtual environments.

However, the pyo3 documentation referenced above was not really helpful in clarifying what was required to configure pyo3 to use a specific virutal env at runtime. That could be a potential enhancement. The documentaiton mentions LD_LIBRARY_PATH for unix systems, and PATH for windows, but PATH really seems to be to only variable that is actually required at runtime, even on unix (I'm using Manjaro Linux). Also, the use-case for the mentioned PYO3_PYTHON variable is still unclear to me. My inital understand was that it intended to be used at runtime to point to a python environment, but now I think it seems to be used at build time, though I am not really sure. I don't think the documentation is really clear on that either :thinking:

By modifying the PATH variable it is indeed possible to configure which virtualenv to use, at runtime, from the Rust program that embeds python. I've updated the reproduction repository with a working (but crude) example. Modifying environment variables from Rust requires a single unsafe call on unix though, so that's something to be aware of.

[phlpphns]

First, thank you @jakob-lilliemarck .

I did a few tests building on your report. My main finding is that this is partly working on Linux (I used WSL with an Ubuntu 24.04 for testing), unfortunately not under windows. -> One can indeed change the virtual environment that is used. -> We can check this by attempting to import a library (e.g. numpy) that is only installed in a part of the venvs. -> What does not appear to be working is to change the interpreter (i.e. selecting another version); more below. -> On Windows, this appears to not be working at all (here I tried various forms of pyenv-win and conda, more for another post)

WHAT DID I DO?

Create a couple of venvs under an wsl ubuntu 24.04. By standard it has python 3.12 only.

Add a possibility to also use "older" python interpreters by adding a suited ppa:

https://askubuntu.com/questions/1398568/installing-python-who-is-deadsnakes-and-why-should-i-trust-them

sudo apt install python3-virtualenv
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.10
sudo apt install python3.10-dev

Create the venvs and install numpy in some of them.

python -m venv venv_test
python -m venv venv_test_2

virtualenv -p python3.10 venv_linux_2_py310/

...

then: source XX/bin/activate pip install numpy

====================================================================

do a "cargo run" on the following code:

Cargo.toml

[package]
name = "pyo3-python-env-ignored"
version = "0.1.0"
edition = "2024"

[dependencies]
dotenv = "0.15.0"
pyo3 = { version = "0.24", features=["auto-initialize"] }
which = "7.0.2"

with an .env file, where the first uncommented entry will be chosen

.env

#VENV_BIN="./uv_venv_global_linux/bin"
#VENV_BIN="./venv_test/bin"
VENV_BIN="./venv_test_2/bin"
VENV_BIN="./venv_linux_2_py310/bin"

src/main.rs

use pyo3::prelude::*;
use std::env;


pub fn ping() {
    Python::with_gil(|py| {
        let sys = PyModule::import(py, "sys").unwrap();

	let _return = py.run(pyo3::ffi::c_str!("print('hallo loaded'); import numpy; print('hallo loaded'); print(numpy.array([3, 2, 1])); print('hallo loaded');"), None, None);
	println!("{_return:?}");

    let _ = py.run(pyo3::ffi::c_str!(r#"
import sys
print(sys.executable, sys.path)
"#), None, None).unwrap();

        let py_path: String = sys.getattr("executable").unwrap().extract().unwrap();
        let py_version: String = sys.getattr("version").unwrap().extract().unwrap();

        println!("executable: {}\nversion: {}", py_path, py_version);
    });
}

fn main() {
    println!("here we go.");
    dotenv::dotenv().ok();

    let venv_bin_relative_path = env::var("VENV_BIN").expect("VENV_BIN relative path must be set");
    println!("{venv_bin_relative_path}");

    let current_dir = env::current_dir().expect("Failed to get current directory");
    println!("\n\n{current_dir:?}");
    
    let venv_bin_path = current_dir.join(venv_bin_relative_path);
    println!("\n\n{venv_bin_path:?}");

    let venv_bin_path_str = venv_bin_path.to_str().expect("Failed to convert path to string");
    println!("\n\n{venv_bin_path_str}");

    let current_path = env::var("PATH").unwrap_or_else(|_| String::new());
    println!("\n\n{current_path}");

    let new_path = format!("{}:{}", venv_bin_path_str, current_path);
    println!("\n\n{new_path}\n\n\n\n\n\n");

    unsafe {
        env::set_var("PATH", &new_path);
    }

    ping();
}

====================================================================

FINDINGS:

Depending on the python environment that was activated either py312 or py310 becomes (and stays) active. This is apparently set during compilation.

hallo loaded
Err(PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'numpy'"), traceback: Some("Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n") })
/mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/./venv_test/bin/python3 ['/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload']
executable: /mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/./venv_test/bin/python3
version: 3.10.16 (main, Dec  4 2024, 08:53:38) [GCC 13.2.0]

hallo loaded
hallo loaded
[3 2 1]
hallo loaded
Ok(())
/mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/./venv_linux_2_py310/bin/python3 ['/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_linux_2_py310/lib/python3.10/site-packages']
executable: /mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/./venv_linux_2_py310/bin/python3
version: 3.10.16 (main, Dec  4 2024, 08:53:38) [GCC 13.2.0]

hallo loaded
Err(PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'numpy'"), traceback: Some("Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n") })
/mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_linux_2_py310/bin/python3 ['/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload']
executable: /mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_linux_2_py310/bin/python3
version: 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0]

hallo loaded
hallo loaded
[3 2 1]
hallo loaded
Ok(())
/mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_test/bin/python3 ['/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_test/lib/python3.12/site-packages']
executable: /mnt/c/Users/USER/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_test/bin/python3
version: 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0]

My interpretation is that after cargo run the interpreter is set (or perhaps only its version? which also would be an achievement already.) Something that I could not yet fully rationalise is that it still prints e.g. version 3.12 if a 3.10 venv is set in the .env file. For example

VENV_BIN="./venv_linux_2_py310/bin" gives

/mnt/c/Users/pkv190/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_linux_2_py310/bin/python3 ['/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload']
executable: /mnt/c/Users/pkv190/SOFTWARE_DEVELOPMENT/RUST/pyo3-python-env-ignored/venv_linux_2_py310/bin/python3
version: 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0]

The correct py_path but something unreasonable about the py_version then. Does that mean that it falls back to the version that was set during compilation if the environment set in the .env file is not sound, but it mixes up things.?

====================================================================

This does not run with uv (neither on windows nor on linux) where I just for sake of separation installed a python 3.11 (I plan to bring this issue in the uv forum; it might be a PATH or environment variables problem).

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv uv_venv_global_linux --python-preference only-managed

source uv_venv_global_linux/bin/activate
cargo run

target/debug/pyo3-python-env-ignored: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

And on Windows for a similar procedure:

error: process didn't exit successfully: target\debug\pyo3_dll_not_found.exe (exit code: 0xc0000135, STATUS_DLL_NOT_FOUND)

[phlpphns]

Hi, the easiest solution is to always activate your venv before running cargo build/run.

Once pyo3 is compiled you cannot change anything from your main.rs, the only thing you can try is doing all your things in a build.rs script.

If you don't want to activate your venv:

I tried something with the PYO3_CONFIG_FILE env variable, it compiles successfully but I can't run it (it fails). The idea is to create a config file:

implementation=CPython
version=3.13
shared=true
abi3=false
lib_name=python3.13
lib_dir=/home/USER/path/to/lib/cpython-3.13.1-linux-x86_64-gnu/lib
executable=/home/USER/path/to/pyo3-python-env-ignored/.venv/bin/python
pointer_width=64
build_flags=
suppress_build_script_link_lines=false

and pass it while running cargo build:

PYO3_CONFIG_FILE=$(pwd)/pyo3_config_file cargo build

It should build successfully but then when i run it, I get:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ec07a612180 (most recent call first):
  <no Python frame>

This behaviour is mentioned here. I guess it might still be valid, figuring out the correct settings is not clear to me yet.

https://pyoxidizer.readthedocs.io/en/latest/pyoxidizer_rust_generic_embedding.html#embed-python-with-pyembed

This is because the embedded Python library doesn’t know how to locate the Python standard library. Essentially, the compiled Python library has some hard-coded defaults for where the Python standard library is located and its default logic is to search in those paths. The references to /install are referring to the build environment for the Python distributions.

The quick fix for this is to define the PYTHONPATH environment variable to the location of the Python standard library. e.g.:

$ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt PYTHONPATH=pyembedded/stdlib cargo run
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
hello, world

We still get some warnings. But our embedded Python interpreter does work!

To make these config changes more permanent and to silence the remaining warnings, you’ll need to customize the initialization of the Python interpreter using C APIs like the Python Initialization Configuration APIs. This requires a fair bit of unsafe code.

[Tehsmash]

Hi, I ran into this exact issue while trying to use pyo3 in a uv managed virtual environment.

Checking the configuration by using PY03_PRINT_CONFIG showed that it was correct detecting the virtual environment's python and lib paths correctly but I was still seeing the ModuleNotFoundError: No module named 'encodings' error.

Further debugging / searching lead me to https://github.com/PyO3/pyo3/discussions/3726

Setting only PYTHONHOME correctly fixed this issue for me:

export PYTHONHOME="$(dirname $(dirname $(realpath $(which .venv/bin/python))))"
uv run cargo run --bin stub_gen

[Poseidon-fan]

My inital understand was that it intended to be used at runtime to point to a python environment, but now I think it seems to be used at build time, though I am not really sure.

@jakob-lilliemarck Me too! I hope that once the rust program was built, it should determine which python env to use on the running host, not the compiling machine. But it seems that pyo3 use the same python env config at compiling and running time ???