scipy.misc is missing data files

[lesteve]

Description

scipy.misc.face() fails since it can not find the associated data file. There are other similar functions that are missing data files in scipy.misc: scipy.misc.electrocardiogram() and scipy.misc.ascent()

The package from emscripten-forge does contain the .dat files. but somehow they are not visible by the kernel.

Reproduce

jupyter lite build --XeusPythonEnv.packages=scipy
jupyter lite serve

Open JupyterLite and start a new notebook with a xeus-python kernel.

from scipy.misc import face

arr = face()

This gives an error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/xpython_42/1607471528.py in <cell line: 3>()
      1 from scipy.misc import face
      2 
----> 3 arr = face()

/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/scipy/misc/_common.py in face(gray)
    220     import bz2
    221     import os
--> 222     with open(os.path.join(os.path.dirname(__file__), 'face.dat'), 'rb') as f:
    223         rawdata = f.read()
    224     data = bz2.decompress(rawdata)

FileNotFoundError: [Errno 44] No such file or directory: '/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/scipy/misc/face.dat'

From the screenshot below, you can see that there are no .dat file in the scipy.misc folder:

This does work fine in a Pyodide kernel.

Expected behavior

No error

[martinRenou]

Thanks for opening an issue.

Indeed xeus-python will filter out those files when packing the environment with empack (for reducing the size of what gets downloaded in the page).

If you don't want this behavior, you can provide your own copy of this file: https://github.com/emscripten-forge/recipes/blob/main/empack_config.yaml Adding an entry for scipy:

  scipy:
    include_patterns:
      - pattern: '*.so'
      - pattern: '*.py'
      - pattern: '**/scipy/misc/*.dat'
    exclude_patterns:
      - pattern: '**/tests/**/*.py'
      - pattern: '**/tests/**/*.so'

Then you need to provide your own config file to xeus-python: jupyter lite build --XeusPythonEnv.empack_config="/home/lesteve/path/to/empack_config.yaml"

We should document this properly at some point. But this is not tested properly yet, and the feature is still very much draft (we should be able to extend the default config instead of needing to replace it entirely).

[lesteve]

Thanks a lot, I'll try this!

[lesteve]

Specifying the empack_config file with additional include for scipy worked.

Would it make sense to make a PR to the emscripten-forge/recipes so that the issue is fixed for everyone? The scipy.misc .dat files are about ~2.2MB if that matters.

[martinRenou]

Those .dat files are just some example data, if I understand correctly? It does not prevent the actual scipy API to work?

I yes, I personally believe this should not be fixed in the default config, but we could document how to fix it for people who want to use those example data.

@DerThorsten what do you think?

[lesteve]

I would agree that scipy.misc.face is not part of the core scipy functionality indeed. At the same time, if I import scipy.misc I would expect scipy.misc.face() to work.