maturin icon indicating copy to clipboard operation
maturin copied to clipboard
verified publisher icon PyO3

Since maturin 1.8.0 cannot import cross-compiled wheel on windows - DLL load failed

Open lorenzwegener opened this issue 10 months ago • 5 comments

Bug Description

I have a rust project that builds a extension for python and publishes it as a wheel. I cross-compile the wheel on Linux for windows.

When I build the wheel with maturin versions 1.5.1, 1.6.0, 1.7.0 or 1.7.8 it can be imported in python on windows. When I build the wheel with maturin versions 1.8.0, 1.8.1, 1.8.2, or 1.8.3 the import fails with: ImportError: DLL load failed while importing <project name>: The specified module could not be found. I build the wheel for target=x86_64-pc-windows-gnu.

I am using:

  • Debian bullseye
  • cargo & rustc: 1.85.0
  • Linux python: 3.9.2
  • Windows python: 3.9.13

Your maturin version (maturin --version)

1.8.3

Your Python version (python -V)

3.9.2

Your pip version (pip -V)

20.3.4

What bindings you're using

pyo3

Does cargo build work?

  • [x] Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

  • [x] Yes

Steps to Reproduce

on linux, install maturin 1.7.8:

$ maturin --version maturin 1.7.8 $ RUST_LOG=maturin=debug PYO3_CROSS_LIB_DIR=/mnt/d/Python39/libs/ RUSTFLAGS="-C target-feature=+aes,+avx2" maturin build -m subproject1/extension/Cargo.toml --target x86_64-pc-windows-gnu --release 2025-03-19T05:27:16.339740Z INFO into_build_context: maturin::build_options: close time.busy=28.1µs time.idle=8.58µs 2025-03-19T05:27:16.339816Z DEBUG maturin::project_layout: Using cargo manifest path from command line argument: "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" 2025-03-19T05:27:16.339829Z DEBUG resolve_cargo_metadata: maturin::project_layout: Resolving cargo metadata from "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" 2025-03-19T05:27:16.855901Z INFO resolve_cargo_metadata: maturin::project_layout: close time.busy=516ms time.idle=1.17µs 2025-03-19T05:27:16.861845Z DEBUG maturin::project_layout: Found pyproject.toml at "/home/lwegener/devel/my_project/subproject1/extension/pyproject.toml" 2025-03-19T05:27:16.862322Z DEBUG resolve_cargo_metadata: maturin::project_layout: Resolving cargo metadata from "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" 2025-03-19T05:27:17.396019Z INFO resolve_cargo_metadata: maturin::project_layout: close time.busy=534ms time.idle=1.89µs 2025-03-19T05:27:17.396660Z DEBUG maturin::project_layout: Project layout resolved project_root=/home/lwegener/devel/my_project/subproject1/extension python_dir=/home/lwegener/devel/my_project/subproject1/extension rust_module=/home/lwegener/devel/my_project/subproject1/extension/my_project python_module=/home/lwegener/devel/my_project/subproject1/extension/my_project extension_name=my_project module_name=my_project 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7 ⚠️ Cross-compiling is poorly supported 2025-03-19T05:27:17.452612Z DEBUG build_wheels: maturin::compile: Running env -u CARGO "cargo" "rustc" "--target" "x86_64-pc-windows-gnu" "--message-format" "json-render-diagnostics" "--manifest-path" "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" "--release" "--lib" "--" "-C" "strip=symbols" Compiling pyo3-build-config v0.22.6 Compiling pyo3-macros-backend v0.22.6 Compiling pyo3-ffi v0.22.6 Compiling pyo3 v0.22.6 Compiling pyo3-macros v0.22.6 Compiling date-utils v0.12.0 (registry ktra) Compiling pyo3-polars v0.19.0 Compiling pyo3-log v0.11.0 Compiling subproject2 v0.1.0 (/home/lwegener/devel/my_project/subproject2) Compiling subproject1 v0.6.9 (/home/lwegener/devel/my_project/subproject1) Compiling my-project v0.6.9 (/home/lwegener/devel/my_project/subproject1/extension) Building [=======================> ] 404/405: my-project Finished release profile [optimized] target(s) in 1m 44s 2025-03-19T05:29:02.209332Z INFO build_wheels:warn_missing_py_init: maturin::compile: close time.busy=38.4ms time.idle=2.03µs 2025-03-19T05:29:02.259065Z DEBUG build_wheels: maturin::module_writer: Adding my_project-0.6.9.dist-info/METADATA 2025-03-19T05:29:02.259289Z DEBUG build_wheels: maturin::module_writer: Adding my_project-0.6.9.dist-info/WHEEL 2025-03-19T05:29:02.259490Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding my_project/init.py 2025-03-19T05:29:02.259656Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding my_project/my_project.pyd from /home/lwegener/devel/my_project/target/x86_64-pc-windows-gnu/release/maturin/my_project.dll 2025-03-19T05:29:05.088307Z INFO build_wheels:write_bindings_module: maturin::module_writer: close time.busy=2.83s time.idle=5.35µs 2025-03-19T05:29:05.088359Z DEBUG build_wheels: maturin::module_writer: Adding my_project-0.6.9.dist-info/RECORD 📦 Built wheel for abi3 Python ≥ 3.7 to /home/lwegener/devel/my_project/target/wheels/my_project-0.6.9-cp37-abi3-win_amd64.whl 2025-03-19T05:29:05.100297Z INFO build_wheels: maturin::build_context: close time.busy=108s time.idle=2.95µs

copy wheel to windows, pip install it and import. import succeeds

F:>python Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import my_project

now uninstall maturin and install maturin 1.8.0

$ maturin --version maturin 1.8.3 $ RUST_LOG=maturin=debug PYO3_CROSS_LIB_DIR=/mnt/d/Python39/libs/ RUSTFLAGS="-C target-feature=+aes,+avx2" maturin build -m subproject1/extension/Cargo.toml --target x86_64-pc-windows-gnu --release 2025-03-19T05:24:24.478489Z INFO into_build_context: maturin::build_options: close time.busy=20.2µs time.idle=7.55µs 2025-03-19T05:24:24.478551Z DEBUG maturin::project_layout: Using cargo manifest path from command line argument: "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" 2025-03-19T05:24:24.478567Z DEBUG resolve_cargo_metadata: maturin::project_layout: Resolving cargo metadata from "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" 2025-03-19T05:24:24.975608Z INFO resolve_cargo_metadata: maturin::project_layout: close time.busy=497ms time.idle=4.50µs 2025-03-19T05:24:24.981897Z DEBUG maturin::project_layout: Found pyproject.toml at "/home/lwegener/devel/my_project/subproject1/extension/pyproject.toml" 2025-03-19T05:24:24.982289Z DEBUG resolve_cargo_metadata: maturin::project_layout: Resolving cargo metadata from "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" 2025-03-19T05:24:25.478546Z INFO resolve_cargo_metadata: maturin::project_layout: close time.busy=496ms time.idle=2.79µs 2025-03-19T05:24:25.479017Z DEBUG maturin::project_layout: Project layout resolved project_root=/home/lwegener/devel/my_project/subproject1/extension python_dir=/home/lwegener/devel/my_project/subproject1/extension rust_module=/home/lwegener/devel/my_project/subproject1/extension/my_project python_module=/home/lwegener/devel/my_project/subproject1/extension/my_project extension_name=my_project module_name=my_project 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7 ⚠️ Cross-compiling is poorly supported 2025-03-19T05:24:25.529946Z DEBUG build_wheels: maturin::compile: Running env -u CARGO PYO3_CONFIG_FILE="/home/lwegener/devel/my_project/target/maturin/pyo3-config-x86_64-pc-windows-gnu-3.7.txt" "cargo" "rustc" "--target" "x86_64-pc-windows-gnu" "--message-format" "json-render-diagnostics" "--manifest-path" "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" "--release" "--lib" "--" "-C" "strip=symbols" Compiling pyo3-build-config v0.22.6 Compiling pyo3-macros-backend v0.22.6 Compiling pyo3-ffi v0.22.6 Compiling pyo3 v0.22.6 Compiling pyo3-macros v0.22.6 Compiling date-utils v0.12.0 (registry ktra) Compiling pyo3-polars v0.19.0 Compiling pyo3-log v0.11.0 Compiling subproject2 v0.1.0 (/home/lwegener/devel/my_project/subproject2) Compiling subproject1 v0.6.9 (/home/lwegener/devel/my_project/subproject1) Compiling my-project v0.6.9 (/home/lwegener/devel/my_project/subproject1/extension) Building [=======================> ] 404/405: my-project Finished release profile [optimized] target(s) in 1m 45s 2025-03-19T05:26:11.263498Z INFO build_wheels:warn_missing_py_init: maturin::compile: close time.busy=39.0ms time.idle=3.69µs 2025-03-19T05:26:11.326468Z DEBUG build_wheels: maturin::module_writer: Adding my_project-0.6.9.dist-info/METADATA 2025-03-19T05:26:11.326718Z DEBUG build_wheels: maturin::module_writer: Adding my_project-0.6.9.dist-info/WHEEL 2025-03-19T05:26:11.326925Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding my_project/init.py 2025-03-19T05:26:11.327104Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding my_project/my_project.pyd from /home/lwegener/devel/my_project/target/x86_64-pc-windows-gnu/release/maturin/my_project.dll 2025-03-19T05:26:14.135717Z INFO build_wheels:write_bindings_module: maturin::module_writer: close time.busy=2.81s time.idle=2.61µs 2025-03-19T05:26:14.135774Z DEBUG build_wheels: maturin::module_writer: Adding my_project-0.6.9.dist-info/RECORD 📦 Built wheel for abi3 Python ≥ 3.7 to /home/lwegener/devel/my_project/target/wheels/my_project-0.6.9-cp37-abi3-win_amd64.whl 2025-03-19T05:26:14.148804Z INFO build_wheels: maturin::build_context: close time.busy=109s time.idle=2.49µs

copy wheel to windows, pip install it and import. import succeeds

\F:>python Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import my_project Traceback (most recent call last): File "", line 1, in File "F:\try_pulsar\lib\site-packages\my_project_init_.py", line 1, in from .my_project import * ImportError: DLL load failed while importing my_project: The specified module could not be found.

lorenzwegener avatar Mar 19 '25 05:03 lorenzwegener

1.7.x:

2025-03-19T05:27:17.452612Z DEBUG build_wheels: maturin::compile: Running env -u CARGO "cargo" "rustc" "--target" "x86_64-pc-windows-gnu" "--message-format" "json-render-diagnostics" "--manifest-path" "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" "--release" "--lib" "--" "-C" "strip=symbols"

1.8.x:

2025-03-19T05:24:25.529946Z DEBUG build_wheels: maturin::compile: Running env -u CARGO PYO3_CONFIG_FILE="/home/lwegener/devel/my_project/target/maturin/pyo3-config-x86_64-pc-windows-gnu-3.7.txt" "cargo" "rustc" "--target" "x86_64-pc-windows-gnu" "--message-format" "json-render-diagnostics" "--manifest-path" "/home/lwegener/devel/my_project/subproject1/extension/Cargo.toml" "--release" "--lib" "--" "-C" "strip=symbols"

Looks like it somehow ignored the PYO3_CROSS_LIB_DIR env var.

messense avatar Mar 19 '25 06:03 messense

I tried building the wheel with maturin 1.8.3 without PYO3_CROSS_LIB_DIR set. That fails with error[E0425]: cannot find function prepare_freethreaded_pythonin cratepyo3--> backtester/src/model/mod.rs:608:15 I also tried with

  • PYO3_CROSS_PYTHON_VERSION=3.9,
  • PYO3_PYTHON=python39 and YO3_CROSS_PYTHON_VERSION=3.9 both result in the same build error.

With both the env vars set and PYO3_PRINT_CONFIG=1 I get -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile -- implementation=CPython version=3.7 shared=true abi3=false lib_name=python37 lib_dir=/home/lwegener/devel/my_project/target/x86_64-pc-windows-gnu/release/build/pyo3-ffi-141095c561ca0c01/out/lib pointer_width=64 build_flags= suppress_build_script_link_lines=false

In the pyo3-ffi-141095c561ca0c01 directory I have: libpypy3-c.def python37.def python37.dll.a python38.dll.a python3.def python3.dll.a

I suspect the python38 libs was the result of an experiment.

lorenzwegener avatar Mar 19 '25 09:03 lorenzwegener

Running objdump on the pyd files in the wheels shows that the working wheel imports python3.dll and the broken wheel import python37.dll.

lorenzwegener avatar Mar 19 '25 09:03 lorenzwegener

Can you create a minimal repro public repository? Your normal build log says it's using abi3 but the PYO3_PRINT_CONFIG=1 output shows abi3=false.

messense avatar Mar 19 '25 09:03 messense

I will try to make a minimal public one.

lorenzwegener avatar Mar 19 '25 09:03 lorenzwegener