xeus-python-demo
xeus-python-demo copied to clipboard
jupyterlite
Troubleshooting an environment that won't build
First off, amazing work on this whole project stack -- seriously a game-changer for leveraging WASM and getting around the issues with Pyodide!
I was hoping to give this template a spin as an entry point to making custom JupyterLite/Xeus environments with our package preinstalled.
Unfortunately, even though our package is on conda-forge and pure Python, it's been a bit painful to figure out which of our deps is failing the build.
Issue investigation
I'm trying to get a JupyterLite build with sleap-io. It's on conda-forge as a noarch package. The deps are minimal but here's the relevant bit from meta.yaml:
run:
- python >=3.7
- ffmpeg
- imageio
- imageio-ffmpeg >=0.5.0
- attrs
- pandas
- simplejson
- h5py >=3.8.0
- hdmf
- numpy
- opencv
- pynwb
- ndx-pose <0.2.0
The only change I made in my fork of the xeus-python-demo template was to add the sleap-io package in environment.yml:
name: xeus-python-kernel
channels:
- https://repo.mamba.pm/emscripten-forge
- conda-forge
dependencies:
- python=3.11
- xeus-python
- ipycanvas
- conda-forge::sleap-io
On the micromamba-based runner in the preconfigured GH Actions, I'm getting this error when trying to build it (with jupyter lite build --contents content --output-dir dist):
error libmamba Could not solve for environment specs
The following packages are incompatible
└─ sleap-io is not installable because there are no viable options
├─ sleap-io [0.1.6|0.1.7|0.1.8] would require
│ └─ av, which does not exist (perhaps a missing channel);
└─ sleap-io 0.1.9 would require
└─ ffmpeg, which does not exist (perhaps a missing channel).
critical libmamba Could not solve for environment specs
[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Command '['/home/runner/micromamba-bin/micromamba', 'create', '--yes', '--no-pyc', '--root-prefix', PosixPath('/tmp/tmpkfxuj4ha/env'), '--name', 'xeus-python-kernel', '--platform=emscripten-wasm[32](https://github.com/talmo/sleap-io-notebook/actions/runs/11089260612/job/30810170200#step:5:33)', '-c', 'https://repo.mamba.pm/emscripten-forge', '-c', 'conda-forge', 'python=3.11', 'xeus-python', 'ipycanvas', 'conda-forge::sleap-io', 'emscripten-abi=3.1.45']' returned non-zero exit status 1.
Full conda list of build-env
/home/runner/micromamba-bin/micromamba list -r /home/runner/micromamba -n build-env
List of packages in environment: "/home/runner/micromamba/envs/build-env"
Name Version Build Channel
──────────────────────────────────────────────────────────────────────────────────
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
anyio 4.6.0 pyhd8ed1ab_1 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py312h66e93f0_5 conda-forge
arrow 1.3.0 pyhd8ed1ab_0 conda-forge
attrs 24.2.0 pyh71513ae_0 conda-forge
beautifulsoup4 4.12.3 pyha770c72_0 conda-forge
bleach 6.1.0 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py312h2ec8cdc_2 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
ca-certificates 2024.8.30 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge
cffi 1.17.1 py312h06ac9bb_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
doit 0.36.0 pyhd8ed1ab_0 conda-forge
empack 4.0.2 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
pip 24.2 pyh8b19718_1 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge
platformdirs 4.3.6 pyhd8ed1ab_0 conda-forge
prometheus_client 0.21.0 pyhd8ed1ab_0 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pygments 2.18.0 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.12.6 hc5c86c4_1_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.20.0 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python_abi 3.12 5_cp312 conda-forge
pyyaml 6.0.2 py312h66e93f0_1 conda-forge
pyzmq 26.2.0 py312hbf22597_2 conda-forge
readline 8.2 h8228510_1 conda-forge
referencing 0.35.1 pyhd8ed1ab_0 conda-forge
requests 2.32.3 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rich 13.8.1 pyhd8ed1ab_0 conda-forge
rpds-py 0.20.0 py312h12e396e_1 conda-forge
send2trash 1.8.3 pyh0d859eb_0 conda-forge
setuptools 75.1.0 pyhd8ed1ab_0 conda-forge
shellingham 1.5.4 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sniffio 1.3.1 pyhd8ed1ab_0 conda-forge
soupsieve 2.5 pyhd8ed1ab_1 conda-forge
terminado 0.18.1 pyh0d859eb_0 conda-forge
tinycss2 1.3.0 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h[484](https://github.com/talmo/sleap-io-notebook/actions/runs/11089260612/job/30810170200#step:4:489)5f30_101 conda-forge
tornado 6.4.1 py312h66e93f0_1 conda-forge
traitlets 5.14.3 pyhd8ed1ab_0 conda-forge
typer 0.12.5 pyhd8ed1ab_0 conda-forge
typer-slim 0.12.5 pyhd8ed1ab_0 conda-forge
typer-slim-standard 0.12.5 hd8ed1ab_0 conda-forge
types-python-dateutil 2.9.0.20240906 pyhd8ed1ab_0 conda-forge
typing-extensions 4.12.2 hd8ed1ab_0 conda-forge
typing_extensions 4.12.2 pyha770c72_0 conda-forge
typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge
tzdata 2024a h8827d51_1 conda-forge
uri-template 1.3.0 pyhd8ed1ab_0 conda-forge
urllib3 2.2.3 pyhd8ed1ab_0 conda-forge
webcolors 24.8.0 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 pyhd8ed1ab_2 conda-forge
websocket-client 1.8.0 pyhd8ed1ab_0 conda-forge
wheel 0.44.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zeromq 4.3.5 ha4adb4c_5 conda-forge
zipp 3.20.2 pyhd8ed1ab_0 conda-forge
zstandard 0.23.0 py312hef9b889_1 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
Trying to pin to conda-forge::sleap-io=0.1.9, which I pushed today to drop av as a hard dependency, I get a different package resolution error:
Looking for: ['python=3.11', 'xeus-python', 'ipycanvas', 'conda-forge::sleap-io=0.1.9', 'emscripten-abi=3.1.45']
https://repo.mamba.pm/emscripten-forge/emscripten-wasm32 Using cache
https://repo.mamba.pm/emscripten-forge/noarch Using cache
conda-forge/emscripten-wasm32 Using cache
conda-forge/noarch Using cache
conda-forge/emscripten-wasm32 Using cache
conda-forge/noarch Using cache
Could not solve for environment specs
The following package could not be installed
└─ sleap-io 0.1.9* is uninstallable because it requires
└─ opencv , which does not exist (perhaps a missing channel).
[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Command '['/Users/talmo/mambaforge3/condabin/mamba', 'install', '--yes', '--prefix', PosixPath('/var/folders/qg/bswb6q8x43b0f8gvr4wq21nm0000gn/T/tmpffcb0qls/env/envs/xeus-python-kernel'), '-c', 'https://repo.mamba.pm/emscripten-forge', '-c', 'conda-forge', 'python=3.11', 'xeus-python', 'ipycanvas', 'conda-forge::sleap-io=0.1.9', 'emscripten-abi=3.1.45']' returned non-zero exit status 1.
(Note: this second log is from running locally in my mambaforge environment, which shouldn't matter much here? Unclear...)
Figuring out compatible packages
So far it's clear that at least these packages aren't supported:
If I use mamba repoquery depends -c conda-forge sleap-io --recursive -p noarch, I can see that there are many deps that should be available, but not all:
Name Version Build Channel
────────────────────────────────────────────────────────────────────────────────
sleap-io 0.1.9 pyhd8ed1ab_0 conda-forge/noarch
numpy >>> NOT FOUND <<<
pandas >>> NOT FOUND <<<
attrs 17.2.0 py_1 conda-forge/noarch
python >>> NOT FOUND <<<
imageio 2.13.1 pyhd8ed1ab_0 conda-forge/noarch
pillow >>> NOT FOUND <<<
opencv >>> NOT FOUND <<<
simplejson >>> NOT FOUND <<<
ffmpeg >>> NOT FOUND <<<
hdmf 0.post0.dev1081 py_0 conda-forge/noarch
requests 2.23.0 pyh8c360ce_2 conda-forge/noarch
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge/noarch
idna 2.9 py_1 conda-forge/noarch
chardet 3.0.4 py_1 conda-forge/noarch
urllib3 1.25.9 py_0 conda-forge/noarch
certifi 2022.6.15 pyhd8ed1ab_1 conda-forge/noarch
idna 3.9 pyhd8ed1ab_0 conda-forge/noarch
pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch
__unix >>> NOT FOUND <<<
pyopenssl 24.2.1 pyhd8ed1ab_2 conda-forge/noarch
cryptography >>> NOT FOUND <<<
brotlipy >>> NOT FOUND <<<
scipy >>> NOT FOUND <<<
setuptools 65.3.0 pyhd8ed1ab_1 conda-forge/noarch
six 1.14.0 py_1 conda-forge/noarch
python-dateutil 2.7.0 py_0 conda-forge/noarch
h5py >>> NOT FOUND <<<
ruamel.yaml >>> NOT FOUND <<<
pynwb 0.2.0 py_0 conda-forge/noarch
imageio-ffmpeg 0.5.1 pyhd8ed1ab_0 conda-forge/noarch
ndx-pose 0.1.1 pyhd8ed1ab_0 conda-forge/noarch
hdmf 3.9.0 pyhff52083_0 conda-forge/noarch
jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge/noarch
attrs 24.2.0 pyh71513ae_0 conda-forge/noarch
importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge/noarch
zipp 3.20.2 pyhd8ed1ab_0 conda-forge/noarch
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge/noarch
jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge/noarch
referencing 0.35.1 pyhd8ed1ab_0 conda-forge/noarch
rpds-py >>> NOT FOUND <<<
pynwb 2.8.2 pyhb401068_0 conda-forge/noarch
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge/noarch
six 1.16.0 pyh6c4a22f_0 conda-forge/noarch
hdmf 3.14.4 pyhb401068_0 conda-forge/noarch
Filtering to the ones that say >>> NOT FOUND <<<:
numpy >>> NOT FOUND <<<
pandas >>> NOT FOUND <<<
python >>> NOT FOUND <<<
pillow >>> NOT FOUND <<<
opencv >>> NOT FOUND <<<
simplejson >>> NOT FOUND <<<
ffmpeg >>> NOT FOUND <<<
__unix >>> NOT FOUND <<<
cryptography >>> NOT FOUND <<<
brotlipy >>> NOT FOUND <<<
scipy >>> NOT FOUND <<<
h5py >>> NOT FOUND <<<
ruamel.yaml >>> NOT FOUND <<<
rpds-py >>> NOT FOUND <<<
Confusingly, several of these (numpy, pandas, scipy) are installable in different examples mentioned here and elsewhere.
I've noticed some comments about emscripten-forge not being well supported across the ecosystem, but trying to do a package search anyway with mamba repoquery depends -c https://repo.mamba.pm/emscripten-forge -c conda-forge --recursive -p noarch -p emscripten-wasm32 sleap-io does seem to provide something:
Name Version Build Channel
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
sleap-io 0.1.9 pyhd8ed1ab_0 conda-forge/noarch
numpy 1.25.2 py311he9a3d86_2 repo.mamba.pm/emscripten-forge/emscripten-wasm32
python 3.11.3 h_bbba66e_25_cpython repo.mamba.pm/emscripten-forge/emscripten-wasm32
pip 20.0.2 py_2 conda-forge/noarch
setuptools 65.3.0 pyhd8ed1ab_1 conda-forge/noarch
python 3.12.4 h_e6699c1_25_cpython repo.mamba.pm/emscripten-forge/emscripten-wasm32
emscripten-abi 3.1.58 h267e887_4 repo.mamba.pm/emscripten-forge/noarch
wheel 0.34.2 py_1 conda-forge/noarch
solvable:prereqmarker >>> NOT FOUND <<<
emscripten-abi 3.1.45 h7b187fe_15 repo.mamba.pm/emscripten-forge/noarch
pandas 1.5.3 h5aab7b2_0 repo.mamba.pm/emscripten-forge/emscripten-wasm32
python-dateutil 2.7.0 py_0 conda-forge/noarch
six 1.14.0 py_1 conda-forge/noarch
pytz 2017.3 py_1 conda-forge/noarch
python-tzdata 2020.1 pyh9f0ad1d_0 conda-forge/noarch
attrs 17.2.0 py_1 conda-forge/noarch
imageio 2.13.1 pyhd8ed1ab_0 conda-forge/noarch
pillow 10.0.1 h12c5105_0 repo.mamba.pm/emscripten-forge/emscripten-wasm32
opencv >>> NOT FOUND <<<
simplejson >>> NOT FOUND <<<
ffmpeg >>> NOT FOUND <<<
hdmf 0.post0.dev1081 py_0 conda-forge/noarch
requests 2.23.0 pyh8c360ce_2 conda-forge/noarch
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge/noarch
idna 2.9 py_1 conda-forge/noarch
chardet 3.0.4 py_1 conda-forge/noarch
urllib3 1.25.9 py_0 conda-forge/noarch
certifi 2022.6.15 pyhd8ed1ab_1 conda-forge/noarch
idna 3.9 pyhd8ed1ab_0 conda-forge/noarch
pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch
__unix >>> NOT FOUND <<<
pyopenssl 24.2.1 pyhd8ed1ab_2 conda-forge/noarch
cryptography 42.0.2 py311h1562a1a_3 repo.mamba.pm/emscripten-forge/emscripten-wasm32
cffi 1.15.1 0_pypy39 conda-forge/noarch
python_abi >>> NOT FOUND <<<
python >>> NOT FOUND <<<
pypy3.9 >>> NOT FOUND <<<
brotlipy >>> NOT FOUND <<<
scipy 1.11.1 h6f59b9d_1 repo.mamba.pm/emscripten-forge/emscripten-wasm32
openblas 0.3.23 h6f59b9d_2 repo.mamba.pm/emscripten-forge/emscripten-wasm32
h5py >>> NOT FOUND <<<
ruamel.yaml >>> NOT FOUND <<<
pynwb 0.2.0 py_0 conda-forge/noarch
imageio-ffmpeg 0.5.1 pyhd8ed1ab_0 conda-forge/noarch
ndx-pose 0.1.1 pyhd8ed1ab_0 conda-forge/noarch
hdmf 3.9.0 pyhff52083_0 conda-forge/noarch
numpy 2.0.1 py312h4e66579_0 repo.mamba.pm/emscripten-forge/emscripten-wasm32
pandas 2.2.2 np125py311he4605d8_0 repo.mamba.pm/emscripten-forge/emscripten-wasm32
jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge/noarch
attrs 24.2.0 pyh71513ae_0 conda-forge/noarch
importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge/noarch
zipp 3.20.2 pyhd8ed1ab_0 conda-forge/noarch
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge/noarch
jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge/noarch
referencing 0.35.1 pyhd8ed1ab_0 conda-forge/noarch
rpds-py >>> NOT FOUND <<<
pynwb 2.8.2 pyhb401068_0 conda-forge/noarch
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge/noarch
six 1.16.0 pyh6c4a22f_0 conda-forge/noarch
hdmf 3.14.4 pyhb401068_0 conda-forge/noarch
Which just leaves these as missing:
solvable:prereqmarker >>> NOT FOUND <<<
opencv >>> NOT FOUND <<<
simplejson >>> NOT FOUND <<<
ffmpeg >>> NOT FOUND <<<
__unix >>> NOT FOUND <<<
python_abi >>> NOT FOUND <<<
python >>> NOT FOUND <<<
pypy3.9 >>> NOT FOUND <<<
brotlipy >>> NOT FOUND <<<
h5py >>> NOT FOUND <<<
ruamel.yaml >>> NOT FOUND <<<
rpds-py >>> NOT FOUND <<<
Is this right? I'm surprised h5py wouldn't be available somehow, but I'm not really sure where to go from here...
Questions
- What is the recommended workflow for making a JupyterLite + xeus-python compatible environment?
- Right now it seems I just have to recurse through all my dependencies and dependencies's dependencies checking if they are available either (a) as
noarchonconda-forge, or (b) onemscripten-forge. - It's not clear what the right tool is for checking whether the environment would resolve (
mamba?micromamba?jupyter lite build ...?).
- Is there a centralized effort for porting packages for JupyterLite + xeus-python compatibility?
- I can't seem to find any info on whether this is being worked on for the different packages mentioned above, or what the blockers would be for those so I can understand the timeline/feasibility of packages complex packages like
opencv. - It's also not clear to me if this is a question best suited for
jupyter-xeus/xeus-python,jupyterlite/xeus,jupyterlite/jupyterlite,emscripten-forge/empack,emscripten-forge/recipesor ???
-
The README makes it seem like the only requirement for a new package is being on
conda-forgeoremscripten-forge, but shouldn't it say something about dependencies?
- Right now, the README tells us to just check
beta.mamba.pmto see if the package isnoarch. -
Our package doesn't even appear on
beta.mamba.pmeven though it's on conda-forge. - Even if it did appear, judging from packages that are there (example), I don't see how I would easily check that all of its deps are also
noarch. - Even if I could check that all
conda-forgedeps arenoarch, it's not like it crosschecks withemscripten-forgewhich I'm guessing would work if it's been manually ported? - The investigation workflow above gets close with
mamba repoquery depends ...but it's not clear if that's the way that this whole stack would resolve the environment.
Thanks for the work on this project! We're really excited to be able to use it! 😄
