Undefined Python Symbols on 32bit Windows Build

[planetmarshall]

Summary

cibuildwheel based build fails on 32 bit Windows builds with the following errors -

 *** Building project with Visual Studio 17 2022...
  [1/2] Building CXX object src\CMakeFiles\_pycharls.dir\pycharls.cpp.obj
  [2/2] Linking CXX shared module src\_pycharls.cp310-win32.pyd
  FAILED: src/_pycharls.cp310-win32.pyd 
  cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=src\CMakeFiles\_pycharls.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1437~1.328\bin\HostX64\x64\link.exe /nologo src\CMakeFiles\_pycharls.dir\pycharls.cpp.obj  /out:src\_pycharls.cp310-win32.pyd /implib:src\_pycharls.lib /pdb:src\_pycharls.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO  /INCREMENTAL:NO /LTCG -LIBPATH:C:\Users\runneradmin\.conan2\p\pybin6c4510a89d30a\p\lib   -LIBPATH:C:\Users\runneradmin\.conan2\p\charlf92cb7ed43d30\p\lib   -LIBPATH:C:\Users\runneradmin\.conan2\p\fmt99c02610800b9\p\lib C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonx86.3.10.11\tools\libs\python310.lib  C:\Users\runneradmin\.conan2\p\charlf92cb7ed43d30\p\lib\charls.lib  C:\Users\runneradmin\.conan2\p\fmt99c02610800b9\p\lib\fmt.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib  && cd ."
  LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1437~1.328\bin\HostX64\x64\link.exe /nologo src\CMakeFiles\_pycharls.dir\pycharls.cpp.obj /out:src\_pycharls.cp310-win32.pyd /implib:src\_pycharls.lib /pdb:src\_pycharls.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO /INCREMENTAL:NO /LTCG -LIBPATH:C:\Users\runneradmin\.conan2\p\pybin6c4510a89d30a\p\lib -LIBPATH:C:\Users\runneradmin\.conan2\p\charlf92cb7ed43d30\p\lib -LIBPATH:C:\Users\runneradmin\.conan2\p\fmt99c02610800b9\p\lib C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonx86.3.10.11\tools\libs\python310.lib C:\Users\runneradmin\.conan2\p\charlf92cb7ed43d30\p\lib\charls.lib C:\Users\runneradmin\.conan2\p\fmt99c02610800b9\p\lib\fmt.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST:EMBED,ID=2" failed (exit code 1120) with the following output:
     Creating library src\_pycharls.lib and object src\_pycharls.exp
  pycharls.cpp.obj : error LNK2001: unresolved external symbol __imp_PyThread_tss_create
  pycharls.cpp.obj : error LNK2001: unresolved external symbol __imp_PyUnicode_AsEncodedString
  pycharls.cpp.obj : error LNK2001: unresolved external symbol __imp_PyByteArray_Resize
  pycharls.cpp.obj : error LNK2001: unresolved external symbol __imp_PyBaseObject_Type
  pycharls.cpp.obj : error LNK2001: unresolved external symbol __imp_PyMem_Calloc
  pycharls.cpp.obj : error LNK2001: unresolved external symbol __imp_PyExc_BufferError
...

See https://github.com/planetmarshall/pillow-jpls/actions/runs/7340945463/job/19987797175 for full log

64 bit build succeeds, as do Ubuntu and Macos builds (all architectures). This can be reproduced locally with cibuildwheel.

Steps to Reproduce

git clone https://github.com/planetmarshall/pillow-jpls.git --branch scikit-build-core
set CIBW_BUILD=cp310-win32
cibuildwheel --platform windows

Legacy scikit-build with 32 bit windows succeeds

[planetmarshall]

This looks like it could be another GHA environment issue, similar to #587

scikit-build sets up the correct compilers for the x86 and x86_64 builds, for example, for cp310-amd64

  -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.38.33130/bin/Hostx86/x64/cl.exe - skipped

and for cp310-win32

  -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.38.33130/bin/Hostx86/x86/cl.exe - skipped

Whilst I assume scikit-build-core has no particular logic to set the compiler and is leaving it to CMake. Whilst I understand the behaviour, it is a change from scikit-build and is useful to know when migrating.

[planetmarshall]

Resolved by setting up the Github Runner environment to use the amd64_x86 cross compiler, though this does mean that the 32 bit Windows build needs special setup and you can't use cibuildwheel to do both amd64 and win32 builds on the same job.

See https://github.com/planetmarshall/pillow-jpls/blob/e61ef3d658bde8d93e53413e2840ca5057aede41/.github/workflows/build_deploy.yml for details

[henryiii]

I believe this is only an issue because you are using bash for Windows. I think it does the right thing on a Windows shell (the default).

[LecrisUT]

I believe this is only an issue because you are using bash for Windows. I think it does the right thing on a Windows shell (the default).

I don't think it is in this case because the issues are within cibw where the shell is overwritten

[henryiii]

Different shells on GHA have different environments. Only the normal shell has the MSVC setup, the bash one does not. It doesn't matter what cibuildwheel uses, the environment setup is different. This affects Meson in cibuildwheel too, and it's exactly the reason we now use the normal shell in cibuildwheel's action, instead of just using bash, which would be simpler.

[henryiii]

See https://github.com/pypa/cibuildwheel/issues/1338 which is why we changed this in cibuildwheel to using powershell on Windows.

[LecrisUT]

Oh, so the issue would be that ilammy/msvc-dev-cmd was executed in a different shell and it did not forward all of its environment to a more shell agnostic environment.

Edit: sorry it is confusing, the edit below happened at the same time as the next reply

More clarification, the python libraries are installed in cibw when it calls actions/setup-python, but at that point it still uses the default shell (setup to be bash here). But when actual cibw it uses the other shell (pwsh).

[henryiii]

No, GitHub Actions sets up the correct MSVC for you for both archetectures, but only for Powershell (and probably CMD). If you run msvc-dev-cmd, then that sets up just what you ask for (all shells), and you end up having to select 32bit vs. 64bit.

[henryiii]

If you remove the bash shell selection and the msvc setup, it should "just work" (fingers crossed).

[LecrisUT]

Another more complicated approach on cibw would be to strip out the second part into a javascript action so that it will pick up whatever shell it has been using in the previous step. I saw that design when looking at how to call setup-python from a JavaScript action.