Support Pyodide 0.28 and bump to Python >=3.13

[agriyakhetarpal]

Closes #47 Closes #45

[rth]

Thanks @agriyakhetarpal for taking over the maintenance!

[agriyakhetarpal]

Pyodide 0.26.4 and 0.27.3 fail with a LookupError claiming an unsupported cp437 encoding after being stripped. I've traced this in Pyodide to the fact that the stripped stdlib also excludes the various encoding implementations from python_stdlib/encodings/, and CPython raises this error when it can't find them (as zipfile tries to decode with either a specified encoding or cp437 – and the latter is undefined).

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I have to note that 0.25.1 and 0.27.3 fail with different errors, so it is also likely that changes in the order of the imports, etc., are causing things to fail. Importing encodings before zip file will not help here, as the encoding itself is lost. We'll need a way to avoid stripping out this specific encoding from this folder. I think the way to do this would be to dive deeper into what discovery.js is doing.

[rth]

Regarding encodings I don't remember the exact details, but indeed the overall idea was that they were taking space so utf8 + a few specific ones should be enough to start Python. But maybe indeed the list of minimal required encodings changed.

so it is also likely that changes in the order of the imports, etc., are causing things to fail I think the way to do this would be to dive deeper into what discovery.js is doing.

Yeah sorry part of it might also be relying on internal emscripten APIs so if those parts (in discovery.js) changed that would explain the breakage.

[agriyakhetarpal]

Thanks! I fixed the LookupError, now one more error remains, so we're almost there!

There is no proper error message in the CI logs, but I am able to reproduce the failure locally, which does have a stack trace:

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

Tap to expand full error message

PythonError: Traceback (most recent call last):
  File "/home/pyodide/pyodide_pack_loader.py", line 10, in setup
    await _module.API.loadDynlib(path, bool(is_shared))
pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'JsException',
  __error_address: 15818000
}

Node.js v23.6.0
FAILED

============================================================================== FAILURES ==============================================================================
_______________________________________________________________________ test_all[scikit-learn] _______________________________________________________________________

example_dir = PosixPath('/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/scikit-learn')
tmp_path = PosixPath('/private/var/folders/b3/2bq1m1_50bs4c7305j8vxcqr0000gn/T/pytest-of-agriyakhetarpal/pytest-230/test_all_scikit_learn_0')

    @pytest.mark.parametrize("example_dir", gen_all_examples())
    def test_all(example_dir, tmp_path):
        stdout = StringIO()
        os.chdir(tmp_path)
        # Assumes there is a node_modules in the current directory
        (tmp_path / "node_modules").symlink_to(
            BASE_DIR / "node_modules", target_is_directory=True
        )
        with redirect_stdout(stdout):
>           cli.main(
                example_path=example_dir / "app.py",
                config_path=None,
                verbose=False,
                include_paths=None,
                write_debug_map=True,
            )

/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/test_examples.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/cli.py:248: in main
    runner.run()
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/runners/node.py:39: in run
    subprocess.run(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

I'm not sure how to debug that so far as it's quite cryptic, so any suggestions would be welcome in case you're familiar with it. :)

[rth]

I think I did encounter it the past, but I'm not 100% sure/remember the cause.

It's raised here but in this case the error is misleading, because we are not trying to load file_packager (legacy pre-wheel) files.

Probably something is not quite right with the sequence in which dynamic libraries are loaded (like if something changed in pyodide in that respect). If you don't find it, ask Hood :)

[rth]

Maybe related to https://github.com/pyodide/pyodide/pull/4726, see @hoodmane's comments in the code there. It looks like _module.API.loadDynlib would be calling locateFile after we are expecting it should no longer happen.

I don't know maybe monkeypatching locateFile back to the original value would be enough to make it work?

[agriyakhetarpal]

One mention of file_packager is here: https://github.com/pyodide/micropip/blob/71bc2e3a5eec9c842f466dd0b5944631feb3b96c/tests/test_install.py#L426-L450, and I see that https://github.com/pyodide/pyodide/pull/2027 removed this format, but it's probably still used for shared libraries.

In scikit-learn's case, shared libraries that need to be imported would be OpenBLAS (due to SciPy). I thought that adding SciPy as an explicit dependency would help, but that's not the case, unfortunately – as OpenBLAS is already included.

The error message is here: https://github.com/pyodide/pyodide/blob/0f3e5f029275482882174b20770caf00290ec4f2/src/js/pyodide.ts#L284-L287

---

Edit: ah, thanks, let me take a look at your comments!

[rth]

I see that https://github.com/pyodide/pyodide/pull/2027 removed this format

Nah I think it was working in 2023 and that PR is from 2021. But this part does keep breaking with emscripten or pyodide updates, I last fixed in it https://github.com/pyodide/pyodide-pack/pull/34

[agriyakhetarpal]

In the meantime, I can confirm that pyodide/pyodide#4726 did not have an effect, as it first turned up in 0.26.0a5 and this test passes with Pyodide 0.26.4 locally (I'll confirm that with CI in a moment).

[ryanking13]

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

This generally happens when we try to locate the shared library from the file system but when it fails (if file is not there, or there was an error while loading the library, etc). As Roman said, emscripten will fallback to locateFile when it happens, but locateFile is not the way we want to load the library, hence the error. So patching the locateFile will not work, instead, I think you need to figure out why the library loading failed.

When building pyodide.asm.js, setting PYODIDE_DEBUG_JS=1 (pyodide flag) and -sDYLINK_DEBUG= (emscripten flag, but it is quite noisy) may help you understand what is happening when loading the libraries.

[agriyakhetarpal]

Thanks for the insights, @rth and @ryanking13! I've been able to isolate the issue – it's coming from the fact that OpenBLAS isn't being correctly bundled against Pyodide 0.27.3:

                                        Packing..                                        
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ No ┃ Package                        ┃ All files ┃ .so libs ┃    Size (MB) ┃ Reduction ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│  0 │ stdlib                         │ 554 → 186 │          │  2.29 → 0.55 │    76.1 % │
│  1 │ joblib-1.4.0-py3-none-any.whl  │   51 → 17 │    0 → 0 │  0.17 → 0.03 │    79.3 % │
│  2 │ numpy-2.0.2-cp312-cp312-pyodi… │ 338 → 142 │  19 → 13 │  3.05 → 2.49 │    18.4 % │
│  3 │ openblas-0.3.26.zip            │     1 → 0 │    1 → 0 │  1.91 → 0.00 │   100.0 % │
│  4 │ scikit_learn-1.6.1-cp312-cp31… │ 511 → 183 │  68 → 43 │  6.53 → 3.12 │    52.2 % │
│  5 │ scipy-1.14.1-cp312-cp312-pyod… │ 730 → 446 │ 110 → 84 │ 13.31 → 7.73 │    41.9 % │
│  6 │ threadpoolctl-3.5.0-py3-none-… │     5 → 1 │    0 → 0 │  0.02 → 0.01 │    65.6 % │
└────┴────────────────────────────────┴───────────┴──────────┴──────────────┴───────────┘

As it can be seen above, libopenblas.so isn't being bundled properly. It's reduced by 100% and lost after modification. This does not happen with Pyodide 0.26.4, which keeps OpenBLAS intact as it was being loaded globally.

Later, this shows up here: [DEBUG] Loading dynamic library: /usr/lib/libopenblas.so, global: false, handle: undefined. However, for Pyodide 0.26.4, we have [DEBUG] Loading dynamic library: /usr/lib/libopenblas.so, global: true, handle: undefined – i.e., global: true instead of false.

This manifests later when we try to import extension modules in SciPy that rely on OpenBLAS, such as scipy.integrate._dop (https://github.com/pyodide/pyodide/blob/8bfd7bfd6a1b33dbf00b4fcf160b78c7307819e4/packages/scipy/patches/0010-Link-openblas-with-modules-that-require-f2c.patch#L18-L30).

Here is a brief summary from my extended logging:

Pyodide 0.26.4

[DEBUG-LOADER] Successfully loaded /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so

and

[DEBUG] Library /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so has 1 accessed symbols

and later

[DEBUG] Symbol accessed: /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so.PyInit__dop

Pyodide 0.27.3

[DEBUG-LOADER] Error loading /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!

and in this case, the error message is indeed not helpful; it hides the traceback. To be more specific, the actual error is masked a few layers underneath by this error:

Tap to show error message

Fatal error: PythonError: Traceback (most recent call last):
  File "/lib/python312.zip/_pyodide/_base.py", line 141, in eval_code_async
    return await CodeRunner(source, return_mode=return_mode, quiet_trailing_semicolon=quiet_trailing_semicolon, filename=filename, flags=flags, dont_inherit=dont_inherit, optimize=optimize).compile().run_async(globals, locals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/_pyodide/_base.py", line 130, in run_async
    coroutine = eval(self.code, globals, locals)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<exec>", line 5, in <module>
  File "/lib/python3.12/site-packages/sklearn/__init__.py", line 11, in <module>
    from .base import clone
  File "/lib/python3.12/site-packages/sklearn/base.py", line 12, in <module>
    from .utils._estimator_html_repr import _HTMLDocumentationLinkMixin, estimator_html_repr
  File "/lib/python3.12/site-packages/sklearn/utils/__init__.py", line 8, in <module>
    from ._chunking import gen_batches, gen_even_slices
  File "/lib/python3.12/site-packages/sklearn/utils/_chunking.py", line 6, in <module>
    from ._param_validation import Interval, validate_params
  File "/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 12, in <module>
    from .validation import _is_arraylike_not_scalar
  File "/lib/python3.12/site-packages/sklearn/utils/validation.py", line 14, in <module>
    from ..utils._array_api import _asarray_with_order, _is_numpy_namespace, get_namespace
  File "/lib/python3.12/site-packages/sklearn/utils/_array_api.py", line 10, in <module>
    from .fixes import parse_version
  File "/lib/python3.12/site-packages/sklearn/utils/fixes.py", line 7, in <module>
    import scipy.stats
  File "/lib/python3.12/site-packages/scipy/stats/__init__.py", line 2, in <module>
    from ._stats_py import *
  File "/lib/python3.12/site-packages/scipy/stats/_stats_py.py", line 13, in <module>
    from . import distributions
  File "/lib/python3.12/site-packages/scipy/stats/distributions.py", line 1, in <module>
    from ._distn_infrastructure import rv_discrete, rv_continuous, rv_frozen
  File "/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py", line 13, in <module>
    from scipy import integrate
  File "/lib/python3.12/site-packages/scipy/__init__.py", line 34, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/__init__.py", line 50, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/scipy/integrate/__init__.py", line 4, in <module>
    from ._ode import *
  File "/lib/python3.12/site-packages/scipy/integrate/_ode.py", line 6, in <module>
    from . import _dop
ImportError: dynamic module does not define module export function (PyInit__dop)

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'ImportError',
  __error_address: 60631024
}

I did not need to use debug builds of Pyodide to get these traces so far. I can put together more logging in a new PR after this one through a pyodide pack --verbose option, as it is currently unused.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through https://github.com/pyodide/pyodide/pull/4876 and https://github.com/pyodide/pyodide/pull/4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

[agriyakhetarpal]

🤔 I tried the netcdf4 example I added with Pyodide 0.26.4 and it seems to pass without needing to ignore unloadable dynlibs, which makes me feel that there is something that I'm missing in the equation. I'll mark it as a draft for now till I investigate more about why it fails with 0.27.3.

Thanks for the review in the meantime!

[agriyakhetarpal]

Some more investigation for this particular example (netcdf4) reveals that:

1. the number of accessed modules is getting reduced with Pyodide 0.27.3 in comparison to 0.26.4; i.e., the latter accesses 5 modules instead of 15 for the former. Modules from numpy.fft and numpy.random do not show up in the bundle .sos list for 0.27.3.
2. the order of the accessed modules is incorrect – libhdf5_hl.so in h5py.libs/ shows up at the end of the list, rather than before other h5py-specific modules that rely on it. This disarrangement is likely what is causing file_packager errors when loading, as dependent dynlibs will fail to load until their parent dynlibs are loaded.

bundle-so-list.txt from Pyodide 0.26.4

Tap to expand

/lib/python3.12/site-packages/numpy/core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_internal.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,True
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,True
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False

bundle-so-list.txt with Pyodide 0.27.3

Tap to expand

/lib/python3.12/site-packages/numpy/_core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5_hl.so,False

---

Unfortunately, installing my debug build of Pyodide does not work here for getting extra logging, as I keep encountering function signature mismatches in the installation stage (with micropip) itself...

[ryanking13]

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through https://github.com/pyodide/pyodide/pull/4876 and https://github.com/pyodide/pyodide/pull/4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

[agriyakhetarpal]

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

Sounds good, I'll leave that for exploration in another PR!

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through https://github.com/pyodide/pyodide/pull/4876 and https://github.com/pyodide/pyodide/pull/4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

Yes, I've adjusted for it. I haven't found a solution to how the netCDF4 test can be fixed. It's good that I added such a test, though, as the rest of the tests were passing and we would have merged with broken functionality. The netCDF4 test passes on 0.24.1, so the failure with 0.27.1 is indeed related.

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders? If yes, I can tweak the order of how the libraries are added to the bundled .sos list and we should be good to go. Otherwise, I think it needs more investigation. I didn't find any changes we made upstream that might have changed the order of the library detection/import – just that the libraries are imported locally, which I addressed. Maybe @hoodmane would know about it.

[ryanking13]

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders?

Yes. After we land RPATH in emscripten, the order of the import should be:

1. LD_LIBRARY_PATH (which will be something like /usr/lib:/lib)
2. paths in the RPATH field.

we can adjust the order in RPATH, so I think we have options to choose the load order.

[agriyakhetarpal]

Sounds good! Though, I don't think we should wait for RPATH support to unblock this PR, as that could likely be after Pyodide 0.28 (which will then break compatibility with pyodide-pack again).

[ryanking13]

Sounds good! Though, I don't think we should wait for RPATH support to unblock this PR, as that could likely be after Pyodide 0.28 (which will then break compatibility with pyodide-pack again).

Sure, this PR can go without waiting for the RPATH support.

[hoodmane]

that could likely be after Pyodide 0.28

I don't want to release Pyodide 0.28 without the RPATH support. The Emscripten 4.0.5 linker supports RPATH already. Once we update to Emscripten >= 4.0.5 we can patch the runtime to use it by backporting some version of https://github.com/emscripten-core/emscripten/pull/23872.

[agriyakhetarpal]

Now that we have support for RPATH in the nightlies, I think we can revisit this. I can start testing with Pyodide 0.28.0a3 here; I'll need to read up a bit on how the RPATH support works.

[agriyakhetarpal]

With Pyodide 0.28, the first hurdle is to get stdlib stripping working. We can see that

┏━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ No ┃ Package ┃ All files ┃ .so libs ┃   Size (MB) ┃ Reduction ┃
┡━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│  0 │ stdlib  │   575 → 0 │          │ 2.34 → 0.00 │   100.0 % │
└────┴─────────┴───────────┴──────────┴─────────────┴───────────┘

so all of the modules are lost, and the Python interpreter fails to start.