Can't call Python from Rust with pyenv

[nathanielsimard]

trafficstars

OS: PopOs (Ubuntu) 22.04 System Python: python3 3.10

Steps:

PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.16
pyenv virtualenv 3.9.16 test
pyenv activate test
cargo run

Output:

     Running `target/debug/noos-serve`
target/debug/noos-serve: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

Now some info:

PYO3_PRINT_CONFIG=1 cargo build

Output:

PYO3_PRINT_CONFIG=1 cargo build
   Compiling pyo3-ffi v0.17.3
error: failed to run custom build command for `pyo3-ffi v0.17.3`

Caused by:
  process didn't exit successfully: `/home/nathaniel/Projects/noos/monoos/noos-serve/target/debug/build/pyo3-ffi-7798a5823d368cc3/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
  cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.9
  shared=true
  abi3=false
  lib_name=python3.9
  lib_dir=/home/nathaniel/.pyenv/versions/3.9.16/lib
  executable=/home/nathaniel/.pyenv/versions/test/bin/python
  pointer_width=64
  build_flags=
  suppress_build_script_link_lines=false

[nathanielsimard]

I resolved the issue by adding the lib directory of the virtualenv:

export LD_LIBRARY_PATH=/home/nathaniel/.pyenv/versions/3.9.16/lib:$LD_LIBRARY_PATH

It took me too much time to figure this out, this could be better documented in a getting started section.

[adamreichold]

Since building works, isn't this a problem with pyenv not adjusting LD_LIBRARY_PATH but still expecting programs linking against those libraries to work? Or is there an expectation that -rpath or something like it is used? Meaning that an extension written in C or C++ should have the same problem actually loading?

[nathanielsimard]

It might be an issue with pyenv, I expected it to handle such things. I'm not sure how extensions in C/C++ behave in that situation, but if they require python-dev, it wouldn't be surprising if the same issue arises.

[mert-kurttutan]

@nathanielsimard Isn't PYTHON_CONFIGURE_OPTS="--enable-shared" supposed to resolve this problem? Link: https://github.com/pyenv/pyenv/wiki#how-to-build-cpython-with---enable-shared

Also, in the tutorial page, there is a relevant section suggesting to use exactly your setting for which the problem still occurs. Link: https://pyo3.rs/v0.18.1/getting_started#virtualenvs

[dashdeckers]

I have the same issue, in fact I am experiencing issues in general with Py03 not calling the right Python. Basically I am trying to make Py03 use the correct virtual environment, and of course use the correct Python executable, but I am getting the weirdest behavior.

In my Rust project I have something along the lines of:

Python::with_gil(|py| {
    let sys = py.import("sys")?;
    let version: String = sys.getattr("version")?.extract()?;
    let path: String = sys.getattr("executable")?.extract()?;
    println!("PYTHON VERSION: {version}");
    println!("PYTHON EXECUTABLE: {path}");
})

And my Cargo.toml has the following:

pyo3 = { version = "0.20.0", features = ["auto-initialize"] }

If I do this:

env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.11:latest
pyenv virtualenv 3.11 envname1
pyenv local envname1
cargo run 

Note: Here it installed 3.11.1

What I get is the following:

target/debug/graph_rl: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

But now if I do this:

env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.10:latest
pyenv virtualenv 3.10 envname2
pyenv local envname2
cargo run 

Note: Here it installed 3.10.9 (already kind of weird?)

What I get is the following:

PYTHON VERSION: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
PYTHON EXECUTABLE: /home/dashdeckers/.pyenv/shims/python3

Which is definitely my system python, because for one the versions don't match and the date does not make sense, and of course because when I remove the .python-version file and execute python3, or python3.10 in my terminal I get the following:

Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

[sashakharina]

@nathanielsimard Your solution has helped me, thanks! But I agree, it's strange that pyenv doesn't manage this.

[GComitini]

I can confirm @dashdeckers's report. pyo3 does not honor pyenv's virtual environment .

On Debian 12 (system Python: 3.11.2), installing a Python 3.12.2 virtual environment with

PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.12.2
pyenv virtualenv 3.12.2 testenv

and then creating a pyo3 test project which should use that virtual environment via

cargo new pyo3test
cd pyo3test
cargo add pyo3 --features auto-initialize
pyenv local testenv

with src/main.rs

use pyo3::prelude::*;

fn main() -> PyResult<()> {
    Python::with_gil(|py| {
        let sys = py.import_bound("sys")?;
        let version: String = sys.getattr("version")?.extract()?;
        let prefix: String = sys.getattr("prefix")?.extract()?;
        println!("PYTHON VERSION: {version}");
        println!("PYTHON PREFIX: {prefix}");

        let os = py.import_bound("os")?;
        let environ = os.getattr("environ")?;
        let pyenv_root = environ.getattr("get")?.call(("PYENV_ROOT",),None)?;
        let pyenv_version = environ.getattr("get")?.call(("PYENV_VERSION",),None)?;
        println!("PYENV_ROOT: {pyenv_root}");
        println!("PYENV_VERSION: {pyenv_version}");

        Ok(())
    })
}

crashes on cargo run with error

target/debug/pyo3test: error while loading shared libraries: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

This happens because $HOME/.pyenv/versions/testenv/lib does not contain libpython3.12.so.1.0.

Setting LD_LIBRARY_PATH to testenv's parent's lib dir, $HOME/.pyenv/versions/3.12.2/lib, which DOES contain libpython3.12.so.1.0, prevents the program from crashing, but yields

PYTHON VERSION: 3.12.2 (main, Apr  4 2024, 09:31:21) [GCC 12.2.0]
PYTHON PREFIX: /home/user/.pyenv/versions/3.12.2
PYENV_ROOT: /home/user/.pyenv
PYENV_VERSION: None

This must be compared with

PYTHON VERSION: 3.12.2 (main, Apr  4 2024, 09:31:21) [GCC 12.2.0]
PYTHON PREFIX: /home/user/.pyenv/versions/testenv
PYENV_ROOT: /home/user/.pyenv
PYENV_VERSION: testenv

which is obtained by running the python script

import sys
print(f'PYTHON VERSION: {sys.version}')
print(f'PYTHON PREFIX: {sys.prefix}')

import os
print(f'PYENV_ROOT: {os.environ.get("PYENV_ROOT")}')
print(f'PYENV_VERSION: {os.environ.get("PYENV_VERSION")}')

in the same directory.

Note that setting the virtual environment with pyenv global instead of pyenv local, as well as soft-linking libpython* from $HOME/.pyenv/versions/3.12.2/lib to $HOME/.pyenv/versions/testenv/lib and setting LD_LIBRARY_PATH to the latter, results in the same behavior. Thus it looks like pyo3 simply runs the python binary hardcoded in libpython3.12.so.1.0 (see e.g. objdump -x $HOME/.pyenv/versions/3.12.2/lib/libpython3.12.so.1.0 | grep RUNPATH ), not the one selected by pyenv.

[jreuben11]

I can confirm that PyO3 does not respect PyEnv virtual environments when calling Python from Rust

pyenv virtualenv 3.11.7 Py03
pyenv local Py03

cargo run --> target/debug/quickstart_p2r: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

adding

export LD_LIBRARY_PATH=${HOME}/.pyenv/versions/3.11.7/lib:$LD_LIBRARY_PATH

is a quick and dirty fix

[GComitini]

Hi @jreuben11. Based on what I wrote before (https://github.com/PyO3/pyo3/issues/2803#issuecomment-2036942279), exporting LD_LIBRARY_PATH is not a fix. If you export LD_LIBRARY_PATH=${HOME}/.pyenv/versions/$VERSION/lib, the loading error is surely gone, but pyo3 uses $VERSION as the python version, which is not the pyenv virtual environment you have configured, but rather the python version upon which the virtual environment is based. If you don't want to go through the test I proposed in the referenced comment, from inside pyo3, try to import a package installed in the virtual environment which is not available to its base version. The import will fail.

If you try this please let me know what happens. Thank you.