PyO3 Fails to Import DLL in Python Virtual Environment on Windows

[luoshuijs]

Bug Description

When running cargo test within a python virtual environment on Windows, PyO3 fails to import DLLs, resulting in a STATUS_DLL_NOT_FOUND error. This occurs even after attempting to use --no-default-features to exclude potential issues caused by default feature sets.

Additional Attempt to Resolve: Following the suggestion from the PyO3 FAQ, adding the directory containing the Python DLL (from the existing Python environment) to the PATH environment variable was attempted. However, this leads to importing modules from the existing Python environment instead of the virtual environment.

Steps to Reproduce

1. Set up a python virtual environment on Windows.

2. Create a minimal Rust project using PyO3.

3. Add a simple test case in src/lib.rs, for example:

    #[test]
    fn test() {
        pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
            let _numpy = PyModule::import(py, "numpy").unwrap();
        });
    }

4. Run the project using cargo test.

5. Observe the error output.

Backtrace

No response

Your operating system and version

Win 10

Your Python version (python --version)

Python 3.11.0

Your Rust version (rustc --version)

rustc 1.71.0-nightly (4a59ba4d5 2023-05-12)

Your PyO3 version

0.19.2

How did you install python? Did you use a virtualenv?

python.org install python -v venv venv

Additional Info

No response

[luoshuijs]

I found that this issue is similar to #1896. I also found that the above steps work normally on Linux (with Ubuntu 22.1 as the system) when running tests with --no-default-features as a parameter. Could this possibly be an issue unique to Windows systems and Python?

[davidhewitt]

This is almost certainly a duplicate of #1896. It's on my list to solve eventually, I think it'll get easier when we can drop 3.7 support and adopt the new PyConfig APIs. @alex - how is the cryptography situation with 3.7 looking? In pydantic we will be dropping 3.7 in our 2.6 release (so probably by end of year).

@luoshuijs, for now you should be able to workaround this by using PYTHONHOME and PYTHONPATH environment variables.

[alex]

Quite bad :-( It's still something like 1/4 of all of our downloads. From the metrics, a huge portion of the downloads comes from Amazon Linux 2.

[luoshuijs]

Thank you very much, @davidhewitt, for your response. However, after trying to set the PYTHONHOME and PYTHONPATH environment variables as you suggested, I'm still facing issues. Here's what I did in PowerShell:.

PS K:\PycharmProjects\python_genshin_artifact> venv/Scripts/activate.ps1
(venv) PS K:\PycharmProjects\python_genshin_artifact>
(venv) PS K:\PycharmProjects\python_genshin_artifact> $env:PYTHONHOME="C:\Users\luoshuijs\AppData\Local\Programs\Python\Python311"
(venv) PS K:\PycharmProjects\python_genshin_artifact> $env:PYTHONPATH="C:\Users\luoshuijs\AppData\Local\Programs\Python\Python311"
(venv) PS K:\PycharmProjects\python_genshin_artifact> $env:PYTHONHOME                                                             
C:\Users\luoshuijs\AppData\Local\Programs\Python\Python311
(venv) PS K:\PycharmProjects\python_genshin_artifact> $env:PYTHONPATH                                                             
C:\Users\luoshuijs\AppData\Local\Programs\Python\Python311
(venv) PS K:\PycharmProjects\python_genshin_artifact> cargo test                                                                  
    Finished test [unoptimized + debuginfo] target(s) in 0.27s
     Running unittests src\lib.rs (target\debug\deps\_python_genshin_artifact-9ae2f0247525dfea.exe)
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `K:\PycharmProjects\python_genshin_artifact\target\debug\deps\_python_genshin_artifact-9ae2f0247525dfea.exe` (exit code: 0xc0000135, STATUS_DLL_NOT_FOUND)

[davidhewitt]

@luoshuijs did you sat PATH too? I think you'll still need that also.

[luoshuijs]

@luoshuijs did you sat PATH too? I think you'll still need that also.

Indeed, I forgot to set the Path. After making the modification, it did have an effect. However, the corresponding modules were imported from the existing Python environment (not a virtual environment), but it's not a big issue. I plan to write a script to resolve the above issue. I am very grateful for your response. I hope to completely fix this problem by the end of this year.

[Simon-Bertrand]

Hello, Any news on this subject?

I found the same problem on Windows with a venv. I got the problem when I tried to compile Torch code using the Rust binding (https://github.com/LaurentMazare/tch-rs) .

The compilation is fine, but when I tried to import the pyd file (just a copy of the compiled DLL) directly into Python, I got a DLL loading error. I used (https://github.com/lucasg/Dependencies) to debug my DLL and discovered that some DLLs were not found (torch_python.dll, torch_cpu.dll, c10.dll) which are in fact the DLLs of the locally installed "torch" Python library. I could find these weirdos files in my venv under the folder ".venv/Lib/site-packages/torch/lib".

A really ugly solution was to use this line before importing the .pyd file into Python.

import os
os.add_dll_directory(Path(os.getcwd()).joinpath(".venv/Lib/site-packages/torch/lib").resolve())

Now all's well.

It seems that PyO3 doesn't link the right DLL directories on Windows from the package installed in the venv. Maybe it's looking for DLLs globally or something else. A friend of mine had the same issue with Pybind and C++, so maybe it's not PyO3 related, I really don't know.

[davidhewitt]

https://github.com/PyO3/pyo3/issues/3589#issuecomment-1820456435 still applies. I'd love to solve this soon, but my own priorities for time have been elsewhere.

[Simon-Bertrand]

#3589 (comment) still applies. I'd love to solve this soon, but my own priorities for time have been elsewhere.

Okey thank you I completely understand, the best workaround I found was to use Linux directly and avoid this crappy Windows.