wjakob
Create Unit tests for Pyodide
Hello, as I mentioned in #687 I am creating unit tests for Pyodide, however, I am creating as a draft because I am going to actually need some assistance as I don't know very well this code base.
Currently I am going trough the route of pybind11 with a pyproject.toml, did some changes in the Cmake file of nanobind in the tests, but for some reason, every time I try to build the tests, I get this error:
FAILED: test_ndarray_ext.pyi /tmp/tmpnfzv8k08/build/test_ndarray_ext.pyi
cd /tmp/tmpnfzv8k08/build && /tmp/build-env-1tz5s2os/bin/python /home/leandrobg/Builds/nanobind/src/stubgen.py -q -i /tmp/tmpnfzv8k08/build -m test_ndarray_ext -o test_ndarray_ext.pyi
Traceback (most recent call last):
File "/home/leandrobg/Builds/nanobind/src/stubgen.py", line 1411, in <module>
main()
File "/home/leandrobg/Builds/nanobind/src/stubgen.py", line 1344, in main
mod_imported = importlib.import_module(mod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'test_ndarray_ext'
[47/83] Generating test_classes_ext.pyi
FAILED: test_classes_ext.pyi /tmp/tmpnfzv8k08/build/test_classes_ext.pyi
cd /tmp/tmpnfzv8k08/build && /tmp/build-env-1tz5s2os/bin/python /home/leandrobg/Builds/nanobind/src/stubgen.py -q -i /tmp/tmpnfzv8k08/build -m test_classes_ext -o test_classes_ext.pyi
Traceback (most recent call last):
File "/home/leandrobg/Builds/nanobind/src/stubgen.py", line 1411, in <module>
main()
File "/home/leandrobg/Builds/nanobind/src/stubgen.py", line 1344, in main
mod_imported = importlib.import_module(mod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'test_classes_ext'
[48/83] Generating test_functions_ext.pyi
FAILED: test_functions_ext.pyi /tmp/tmpnfzv8k08/build/test_functions_ext.pyi
cd /tmp/tmpnfzv8k08/build && /tmp/build-env-1tz5s2os/bin/python /home/leandrobg/Builds/nanobind/src/stubgen.py -q -i /tmp/tmpnfzv8k08/build -m test_functions_ext -o test_functions_ext.pyi
Traceback (most recent call last):
File "/home/leandrobg/Builds/nanobind/src/stubgen.py", line 1411, in <module>
main()
File "/home/leandrobg/Builds/nanobind/src/stubgen.py", line 1344, in main
mod_imported = importlib.import_module(mod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'test_functions_ext'
Which, I don't know if I am doing something wrong, or a bug, because I don't see anything special related to Pyodide/Emscripten in the Cmake test in pybind11
@leandro-benedet-garcia Could you give me access to push to your fork? I've got this approach working, at least locally. Or you can just pull the changes yourself from https://github.com/henryiii/nanobind @ henryiii/fix/standalonetests.
@henryiii I added you to the fork, I tried with just pull request but in the CI it showed the same error.
Ahhh, good point, I can just run the CI on my fork first. The problem you are seeing is that you can't import the modules you are building during the compilation phase when cross-compiling. Only later via WASM. So we need to be able to disable the subgen tests. (Well, easily, anyway. There are ways to set the cross-compiling emulator in CMake)
Allrighty, since it seems to be working now I will put it for review, thank you @henryiii
Let me know if you have questions about any of the changes! Very high level, I made the find_package(Python) GLOBAL, since it was not usable outside the directory it's triggered from, but the nanobind_* functions are. I changed Python_INCLUDE_DIRS into using Python::Module's INTERFACE_INCLUDE_DIRS entirely to get an error if it's not visible in that scope; before it would just leave it empty. Making a FATAL_ERROR if Python_INCLUDE_DIRS is undefined would do the same thing. All path variables in CMake should be surrounded by quotes, I noticed them missing in various places, but didn't fix them all for now. I added -fexceptions if compiling for Emscripten, since it's required (and pybind11 does that now too). Tests support being built directly standalone. The stubs are not generated if cross-compiling. I've disabled the stub tests on the emscripten platform. And the stub tests now look up the stubs relative to the compiled modules, rather than the tests.
If you want to try out out locally without Emscripten, you can do something like this:
pipx run build --wheel
uv venv
uv pip install dist/nanobind_tests-0.0.1-cp312-cp312-macosx_14_0_x86_64.whl
py -m pytest
What's the status of this PR? Anything left to do, should I review?
it is ready for review.
For emscripten, you have to build the test module as a wheel and then test it. There are two ways to do that. In pybind11, we make the test module buildable. The other option would be to have a way to enable tests to be build as the wheel in the outer pyproject.toml. Names can't be changed, so you couldn't have both at the same time, but I think you could use the tests/pyproject.toml but the main CMakeLists.txt - would have to investigate. This would require fewer changes, but I was just updating this PR, which started with the test module buildable (and that does mimic pybind11. But in pybind11 we haven't moved to scikit-build-core yet for the outer build).
I can try the other method in a couple of days.
Any progress on restructuring the PR? Should I close it for now?
I will try my hand at it this weekend.
I will close this PR for now, though I remain open to shipping such a CI target if the integration is done less intrusively.