PyInit for cython extension not located when compiled with pyodide, but is in pyodide auditwheel output

[AndrewAnnex]

trafficstars

I am transitioning my library to use scikit-build-core to enable me to provide Pyodide wheels for my python project that provides both a ctypes based interface to a pure c library and a cython extension that links to the same shared library (for important reasons).

My project is structured like so:

src/
      spiceypy/ 
             __init__.py (imports functions from spiceypy.py and imports cyice to be a sub-namespace spiceypy.cyice)
             spiceypy.py (contains all the ctypes wrapper functions)
             utils/
                       __init__.py 
                       libcspice.wasm (shared library I wrap with ctypes and also link to the cython extension
              cyice/
                       __init__.py 
                       cyice.so (the pyx and pxd files are also in here as is the generated cython c code)
     

I want to keep this layout to minify any api breaking changes, and so far as I can tell should work.

Because of the complicated layout I built this prototype python package to verify I could get things working without too many extra tests or complications, and it does work: https://github.com/AndrewAnnex/spiceypy-cyice-scikit-build-core-test

However in the main project my tests fail to import this cython extension, claiming that the PyInit_cyice function is not exported

2025-09-29T22:28:13.1982434Z __________________ ERROR collecting benchmarks/test_cyice.py ___________________
2025-09-29T22:28:13.1988102Z ImportError while importing test module '/tmp/cibw-run-9djpqn83/cp312-pyodide_wasm32/venv-test/lib/python3.12/site-packages/spiceypy/benchmarks/test_cyice.py'.
2025-09-29T22:28:13.1989985Z Hint: make sure your test modules/packages have valid Python names.
2025-09-29T22:28:13.1991029Z Traceback:
2025-09-29T22:28:13.1991636Z /lib/python312.zip/importlib/__init__.py:90: in import_module
2025-09-29T22:28:13.1992589Z     return _bootstrap._gcd_import(name[level:], package, level)
2025-09-29T22:28:13.1993216Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-29T22:28:13.1993897Z ../venv-test/lib/python3.12/site-packages/spiceypy/__init__.py:28: in <module>
2025-09-29T22:28:13.1994457Z     from .cyice import cyice
2025-09-29T22:28:13.1994887Z ../venv-test/lib/python3.12/site-packages/spiceypy/cyice/__init__.py:29: in <module>
2025-09-29T22:28:13.1995337Z     from .cyice import *
2025-09-29T22:28:13.1995737Z E   ImportError: dynamic module does not define module export function 

However, running pyodide auditwheel shows it as expected:

spiceypy/cyice/cyice.so:
      FUNC	__wasm_call_ctors
      FUNC	__wasm_apply_data_relocs
      FUNC	PyInit_cyice
    GLOBAL	__pyx_module_is_main_spiceypy__cyice__cyice

I can also directly load the libcspice.wasm with ctypes within pyodide (via jupyterlite) and see it works as expected.

So there seems to something subtly wrong with the cython extension compilation I'm not spotting. Or maybe the relative import is wrong in some way thats causing the export to not work as expected? Directly loading the same cyice.so with ctypes shows that the library seems to replicate the error also, but in my cmake I am as far as I can tell compiling a shared library and I am even now explicitly exporting the pyinit function (https://github.com/AndrewAnnex/SpiceyPy/blob/f295c02d3ca8d6e812f20a694668af7352668848/CMakeLists.txt#L136-L141), but I didn't have to make any hacks like I did in the test repo to have this work as expected.

I also get a weird error from pyodide in the tests claiming "Error: Didn't expect to load any more file_packager files!" which isn't clear, but I get the same error when attempting to install the wheel inside jupyterlite with piplite, but if I run it a 2nd time it goes away, which isn't confidence building.. But I am not sure if that's actually an issue caused by the compilation or something else entirely.

Link to branch: https://github.com/AndrewAnnex/SpiceyPy/tree/add_pyodide_build Link to PR: https://github.com/AndrewAnnex/SpiceyPy/pull/516 Link to my CMakeLists.txt: https://github.com/AndrewAnnex/SpiceyPy/blob/add_pyodide_build/CMakeLists.txt Link to my pyproject.toml: https://github.com/AndrewAnnex/SpiceyPy/blob/add_pyodide_build/pyproject.toml Build log from github actions: build_log_github.txt

[LecrisUT]

• Your library has dependencies, you should invert the order of from import and the ctypes load (but also try to avoid it altogether, there are better more native tools)
• Remove this line the relative navigation is trying to find test_cyice.test_cyice.*, but test_cyice.test_cyice is not a package, it's a module. You could use from . import *, but even that is not advised due to the *, so just hold off adding it until you have something you need to import

[AndrewAnnex]

@LecrisUT I don't fully follow your first point, I am not sure what other tools you mean, but otherwise you are saying the import order matters here. But is it because the cython extension is being loaded before the ctypes call to load the dependency, or the dependency is loaded before the cython extension (and which one is the correct way to load it?). I don't fully know the mechanisms here, but that would seem to imply that essentially loading the cython extension without the dependency pre-loaded causes the PyInit_cyice function to not be found?

For your second point, those lines are in the test repo that are fully functional, I don't get an import error in the ci builds. That said I was confused exactly on how that import is getting resolved as so I can try to mess around with it in my main package's pr. I know it'd be advisable to implement all here, but my cython extension has over 100 functions I believe already, and my main wrapper has over 600 not including useful helper methods and decorators so I've been trying to avoid that for the moment.

[LecrisUT]

@LecrisUT I don't fully follow your first point, I am not sure what other tools you mean, but otherwise you are saying the import order matters here.

I was referring to the target_link_libraries of the python module, or effectively any dynamic library dependency that it would require to link to. If you do the search of those libraries manually via ctypes, than the library that depends on those symbols has to come after it. Otherwise you would have unresolved symbols. Do note that the ctypes loading is unreliable on exotic OS and/or architectures.

but that would seem to imply that essentially loading the cython extension without the dependency pre-loaded causes the PyInit_cyice function to not be found?

red-herring. PyInit_cyice will be found and it would/could fail at some other point. But the initial error message you presented doesn't indicate that PyInit_cyice is not found, but rather it is trying to import something that does not exist, or rather the module path construction is not appropriate in there. Either that or the * is interfering.

For your second point, those lines are in the test repo that are fully functional, I don't get an import error in the ci builds.

As in it works in other environments? Weird, could those just be silent no-ops? The from .cyice cimport * inside the cyice.pyx is definitely misplaced.

I know it'd be advisable to implement all here, but my cython extension has over 100 functions I believe already, and my main wrapper has over 600 not including useful helper methods and decorators so I've been trying to avoid that for the moment.

Then just don't import the individual names. Just import the module and access the functions with a namespace.

[AndrewAnnex]

@LecrisUT getting back to this after a while

I think the error message I posted above was truncated (but preserved in the log files I attached). I don't see how to believe the missing PyInit_cyice is a red herring otherwise, unless you didn't realize that was the actual error message.

Below is a full message from a recent build:

 __________________ ERROR collecting tests/test_spiceerrors.py __________________
  ImportError while importing test module '/tmp/cibw-run-emwxdzhc/cp312-pyodide_wasm32/venv-test/lib/python3.12/site-packages/spiceypy/tests/test_spiceerrors.py'.
  Hint: make sure your test modules/packages have valid Python names.
  Traceback:
  /lib/python312.zip/importlib/__init__.py:90: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ../venv-test/lib/python3.12/site-packages/spiceypy/tests/test_spiceerrors.py:28: in <module>
      from spiceypy import cyice
  ../venv-test/lib/python3.12/site-packages/spiceypy/__init__.py:31: in __getattr__
      module = importlib.import_module(f"{__name__}.cyice")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  /lib/python312.zip/importlib/__init__.py:90: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ../venv-test/lib/python3.12/site-packages/spiceypy/cyice/__init__.py:32: in <module>
      from spiceypy.cyice.cyice import (
  E   ImportError: dynamic module does not define module export function (PyInit_cyice)
  =========================== short test summary info ============================

In any case, I went through and removed my wild card imports, added __all__ lists, but I still get the error listed above.

really having trouble unpacking your advice:

" If you do the search of those libraries manually via ctypes, than the library that depends on those symbols has to come after it."

I have Python library "A" that imports both a pure C library "B" and a cython library "D" where D depends on/imports "B"

I understand then that library "B" needs to be first regardless if "D" or "A" is imported. But first in what? First in the ld_library_path? First thing imported? I believe I am using ctypes to load the pure C library first before the cython library always but I will check again.

[LecrisUT]

Ok, the full traceback helps, can you point to the codebase with that state?