pyo3 icon indicating copy to clipboard operation
pyo3 copied to clipboard

try to link non abi3 lib `python3x.lib` when abi3 enabled

Open trim21 opened this issue 6 months ago • 5 comments

Bug Description

I'm running maturin build --release -i 3.9 pypy3.9 pypy3.10 with maturin 1.8.6, with abi feature enabled:

pyo3 = { version = "0.25.0", features = ["extension-module", "generate-import-lib", 'abi3-py39'] }

but I find it try to link python312.lib instead of python3.lib

  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: LINK : fatal error LNK1181: cannot open input file 'python312.lib'␍

Your maturin version (maturin --version)

maturin 1.8.6

Your Python version (python -V)

both 3.9 or 3.12

Your pip version (pip -V)

pip 24.3.1

What bindings you're using

pyo3

Does cargo build work?

  • [x] Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

there isn't any path in args

  • [x] Yes

Steps to Reproduce

  1. git clone --branch bug https://github.com/trim21/uuid-utils
  2. cd uuid-utils
  3. maturin build --release -i 3.12 pypy3.9 pypy3.10

verbose log: https://gist.github.com/trim21/da254d65ab3f4522ea3103be4af046d4

doesn't look right:

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.9
  shared=true
  abi3=true
  lib_name=python312
  lib_dir=C:\Users\Trim21\proj\uuid-utils\target\release\build\pyo3-ffi-db10177e3af506ff\out\lib
  pointer_width=64
  build_flags=
  suppress_build_script_link_lines=false

trim21 avatar May 17 '25 18:05 trim21

The verbose log shows that you are actually building with pyo3 v0.23.1, have you tried upgrade it to at least 0.23.5 and ensure python3-dll-a is on latest version?

messense avatar May 18 '25 02:05 messense

I upgrade it to 0.25.0 and still:

📦 Including license file "C:\Users\Trim21\proj\uuid-utils\LICENSE.md"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
⚠️  Warning: Failed to determine python platform
🐍 Not using a specific python interpreter (automatically generating windows import library)
📡 Using build options features from pyproject.toml
   Compiling pyo3-build-config v0.25.0
   Compiling pyo3-macros-backend v0.25.0
   Compiling pyo3-ffi v0.25.0
   Compiling pyo3 v0.25.0
   Compiling pyo3-macros v0.25.0
   Compiling uuid-utils v0.10.0 (C:\Users\Trim21\proj\uuid-utils)
error: linking with `link.exe` failed: exit code: 1181
  |
  = note: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\link.exe" "/DEF:C:\\Users\\Trim21\\AppData\\Local\\Temp\\rustcOlf3Tg\\lib.def" "/NOLOGO" "C:\\Users\\Trim21\\AppData\\Local\\Temp\\rustcOlf3Tg\\symbols.o" "<6 object files omitted>" "C:\\Users\\Trim21\\proj\\uuid-utils\\target\\release\\deps/{libuuid-8ac3b9bfb36cbb71.rlib,librand-1dda470bd604d6a6.rlib,librand_chacha-0fabe40789fc55ed.rlib,librand_core-f8058201550edb30.rlib,libgetrandom-77184a3522ae39b0.rlib,libsha1_smol-f4e8f60713d38390.rlib,libmd5-61b4a176159b27c4.rlib,libdigest-ba320fa3c0e225d6.rlib,libblock_buffer-1a46078d05cfc6f8.rlib,libcrypto_common-293bce7df6df2407.rlib,libgeneric_array-2d66856823516b2e.rlib,libtypenum-c814792aa62bf50e.rlib,libatomic-3793744390d0857c.rlib,libbytemuck-1db23b790d032f37.rlib,librand-5db9b3e50205de32.rlib,librand_chacha-e3d4dac0cd184a1b.rlib,libppv_lite86-b4d824469f291b5d.rlib,libzerocopy-ebd732f302d95ec3.rlib,librand_core-4d1e802b6a0fc545.rlib,libgetrandom-433d96fd46a0ae6f.rlib,libcfg_if-9baf0c6e20c59ab1.rlib,libpyo3-6289f76d95cdae20.rlib,libmemoffset-95d5f532bb14342d.rlib,libonce_cell-a95a06a88addb51d.rlib,libpyo3_ffi-8eecc57c23ebfb71.rlib,liblibc-a10769e62fc79919.rlib,libunindent-bf416bdc2c4248b8.rlib,libmac_address-c5a59bec6241cedb.rlib,libwinapi-7b5e0da6eecbcc8d.rlib}.rlib" "<sysroot>\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib/{libstd-*,libpanic_unwind-*,libwindows_targets-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libunwind-*,libcfg_if-*,liballoc-*,librustc_std_workspace_core-*,libcore-*,libcompiler_builtins-*}.rlib" "bcrypt.lib" "advapi32.lib" "python312.lib" "legacy_stdio_definitions.lib" "iphlpapi.lib" "kernel32.lib" "kernel32.lib" "kernel32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/LIBPATH:C:\\Users\\Trim21\\proj\\uuid-utils\\target\\release\\build\\pyo3-ffi-8e4679538faf1563\\out\\lib" "/OUT:C:\\Users\\Trim21\\proj\\uuid-utils\\target\\release\\deps\\uuid_utils.dll" "/OPT:REF,ICF" "/DLL" "/IMPLIB:C:\\Users\\Trim21\\proj\\uuid-utils\\target\\release\\deps\\uuid_utils.dll.lib" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\libstd.natvis"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: LINK : fatal error LNK1181: cannot open input file 'python312.lib'␍


error: could not compile `uuid-utils` (lib) due to 1 previous error
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "C:\\Users\\Trim21\\proj\\uuid-utils\\Cargo.toml" "--release" "--lib"`

with latest "python3-dll-a" version = "0.2.14"

trim21 avatar May 18 '25 02:05 trim21

The .lib file is generated in pyo3 build script, so it's probably something wrong between pyo3-build-config and python3-dll-a, I'll transfer this issue to pyo3.

messense avatar May 18 '25 02:05 messense

Microsoft (R) Library Manager Version 14.29.30159.0 Copyright (C) Microsoft Corporation. All rights reserved.

Creating library C:\Users\Trim21\proj\uuid-utils\target\release\build\pyo3-ffi-db10177e3af506ff\out\lib\python3.lib and object C:\Users\Trim21\proj\uuid-utils\target\release\build\pyo3-ffi-db10177e3af506ff\out\lib\python3.exp

from the verbose log, it's generating python3.lib correctly, but the lib_name=python312 in PYO3_PRINT_CONFIG seems wrong.

messense avatar May 18 '25 02:05 messense

OK I think I know what is happening:

  1. maturin generates a pyo3 config file with abi3=false regardless of abi3-* features (because abi3 is a mode, not a interpreter config)
    implementation=CPython
    version=3.12
    shared=true
    abi3=false
    build_flags=
    suppress_build_script_link_lines=false
    pointer_width=64
    
  2. pyo3-build-config reads the config file and set lib_name=python3.12 because abi3=false: https://github.com/PyO3/pyo3/blob/b4d5935d3fefa908cc8569627bb1ee18ed3d6d60/pyo3-build-config/src/impl_.rs#L549-L559
  3. pyo3-build-config tries to fixup abi3 and abi3 version but that's too late for lib_name: https://github.com/PyO3/pyo3/blob/b4d5935d3fefa908cc8569627bb1ee18ed3d6d60/pyo3-build-config/src/impl_.rs#L457-L463

the fix could be move config.abi3 |= is_abi3(); to https://github.com/PyO3/pyo3/blob/b4d5935d3fefa908cc8569627bb1ee18ed3d6d60/pyo3-build-config/src/impl_.rs#L549

cc @davidhewitt what do you think?

messense avatar May 18 '25 03:05 messense