stlite ModuleNotFoundError: The module 'protobuf' is included in the Pyodide distribution, but it is not installed.

I get this error trace inside the electron app when I run the dmg created by either npm run dist or yarn dist. But before this npm run serve or yarn serve for preview does not result in this and the electron app runs fine.

Error during booting up

Traceback (most recent call last): File "/lib/python311.zip/_pyodide/_base.py", line 573, in eval_code_async await CodeRunner( File "/lib/python311.zip/_pyodide/_base.py", line 393, in run_async coroutine = eval(self.code, globals, locals) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 2, in File "streamlit/init.py", line 69, in File "streamlit/delta_generator.py", line 35, in File "streamlit/cursor.py", line 20, in File "streamlit/runtime/init.py", line 17, in File "streamlit/runtime/runtime.py", line 26, in File "streamlit/proto/BackMsg_pb2.py", line 5, in ModuleNotFoundError: The module 'protobuf' is included in the Pyodide distribution, but it is not installed. You can install it by calling: await micropip.install("protobuf") in Python, or await pyodide.loadPackage("protobuf") in JavaScript See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

And in the pop-up window in the electron app:

A JavaScript error occurred in the main process Uncaught Exception: PythonError: Traceback (most recent call last): File "/lib/python311.zip/_pyodide/_base.py", line 573, in eval_code_async await CodeRunner( File "/lib/python311.zip/_pyodide/_base.py", line 393, in run_async coroutine = eval(self.code, globals, locals) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 2, in File "streamlit/init.py", line 69, in File "streamlit/delta_generator.py", line 35, in File "streamlit/cursor.py", line 20, in File "streamlit/runtime/init.py", line 17, in File "streamlit/runtime/runtime.py", line 26, in File "streamlit/proto/BackMsg_pb2.py", line 5, in ModuleNotFoundError: The module 'protobuf' is included in the Pyodide distribution, but it is not installed. You can install it by calling: await micropip.install("protobuf") in Python, or await pyodide.loadPackage("protobuf") in JavaScript See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

at new_error (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:10014) at wasm://wasm/02257296:wasm-function[300]:0x158e62 at wasm://wasm/02257296:wasm-function[454]:0x15ff9e at _PyEM_TrampolineCall_JS (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:120223) at wasm://wasm/02257296:wasm-function[1083]:0x1a3494 at wasm://wasm/02257296:wasm-function[3413]:0x28a2bd at wasm://wasm/02257296:wasm-function[2063]:0x1e431e at wasm://wasm/02257296:wasm-function[1090]:0x1a397d at wasm://wasm/02257296:wasm-function[1093]:0x1a3c3e at wasm://wasm/02257296:wasm-function[1094]:0x1a3ce0 at wasm://wasm/02257296:wasm-function[3226]:0x2689a4 at wasm://wasm/02257296:wasm-function[3227]:0x26e800 at wasm://wasm/02257296:wasm-function[1096]:0x1a3e08 at wasm://wasm/02257296:wasm-function[1091]:0x1a3a98 at wasm://wasm/02257296:wasm-function[442]:0x15f76b at Module.callPyObjectKwargs (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:64297) at Module.callPyObject (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:65365) at Timeout.wrapper [as _onTimeout] (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:32855) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7)

Im running it on Mac 11.6 node -v && npm -v && yarn -v v18.14.0 9.3.1 1.22.22.

devDependencies": { "@stlite/desktop": "^0.51.0", "cross-env": "^7.0.3", "electron": "^28.2.1", "electron-builder": "^24.9.1"

My project directory contains:

build dist node_modules package.json requirements.txt streamlit_app -> streamlit_app.py yarn.lock

I can see that, /node_modules/pyodide/protobuf-4.23.1-cp311-cp311-emscripten_3_1_46_wasm32.whl is present.

The requirements.txt contains:

st_files_connection PyYAML streamlit-tags

Also my package.json is modified for local file access (which I have modified by running the dump again)

{ // ...other fields... "stlite": { "desktop": { "nodeJsWorker": true, "nodefsMountpoints": { "/mnt": "/Users/Shared/Appfiles/" } } } }

I there is no straight up permanent solution, if someone could suggest a hacky way to package it for now, as npm run serve or yarn serve for preview does not result in this error and the electron app works perfectly well with local directory access as desired, thats why I think its different from #648

Apr 02 '24 16:04 drankush

@whitphx Would appreciate your insight in this issue.

Apr 05 '24 17:04 drankush

Can you show the entire package.json? Does it contain the "build" field as the example?
Can you provide the file tree in the build directory, e.g. the result of tree ./build? Does build/pyodide/protobuf-4.23.1-cp311-cp311-emscripten_3_1_45_wasm32.whl exist?

Apr 08 '24 09:04 whitphx

./build ├── asset-manifest.json ├── c43e2f2e6b6fd378c8e1.module.wasm ├── electron │ ├── main.js │ ├── preload.js │ └── worker.js ├── index.html ├── pyodide │ ├── Jinja2-3.1.2-py3-none-any.whl │ ├── MarkupSafe-2.1.3-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── Pillow-10.0.0-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── PyYAML-6.0.1-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── attrs-23.1.0-py3-none-any.whl │ ├── cachetools-5.3.1-py3-none-any.whl │ ├── cramjam-2.6.2-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── fastparquet-2023.7.0-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── fsspec-2023.6.0-py3-none-any.whl │ ├── jsonschema-4.17.3-py3-none-any.whl │ ├── micropip-0.5.0-py3-none-any.whl │ ├── numpy-1.26.1-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── packaging-23.1-py3-none-any.whl │ ├── pandas-1.5.3-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── protobuf-4.23.1-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── pyodide-lock.json │ ├── pyodide.asm.js │ ├── pyodide.asm.wasm │ ├── pyodide.mjs │ ├── pyodide_http-0.2.1-py3-none-any.whl │ ├── pyrsistent-0.19.3-cp311-cp311-emscripten_3_1_46_wasm32.whl │ ├── python_dateutil-2.8.2-py2.py3-none-any.whl │ ├── python_stdlib.zip │ ├── pytz-2023.3-py2.py3-none-any.whl │ ├── six-1.16.0-py2.py3-none-any.whl │ ├── toolz-0.12.0-py3-none-any.whl │ └── typing_extensions-4.7.1-py3-none-any.whl ├── requirements.txt ├── site-packages-snapshot.tar.gz ├── static │ ├── css │ │ ├── 8050.c24b25fa.chunk.css │ │ ├── 8957.cd5b39ce.chunk.css │ │ ├── 9449.1edfa457.chunk.css │ │ └── main.fb5f4d99.css │ ├── js │ │ ├── 1046.19145034.chunk.js │ │ ├── 1122.9b0e508e.chunk.js │ │ ├── 1189.ff18a565.chunk.js │ │ ├── 1261.e15a45e6.chunk.js │ │ ├── 142.9be80baa.chunk.js │ │ ├── 144.b10fb472.chunk.js │ │ ├── 1579.418b1244.chunk.js │ │ ├── 1729.3e10491c.chunk.js │ │ ├── 1740.c4ec1606.chunk.js │ │ ├── 1839.45bfc04d.chunk.js │ │ ├── 2385.8cbb7ba4.chunk.js │ │ ├── 2448.9a8682bf.chunk.js │ │ ├── 2694.6d6de2d2.chunk.js │ │ ├── 2694.6d6de2d2.chunk.js.LICENSE.txt │ │ ├── 2753.9ea4d84f.chunk.js │ │ ├── 2873.69bb3f79.chunk.js │ │ ├── 2909.692a5576.chunk.js │ │ ├── 2940.b9c9a565.chunk.js │ │ ├── 3208.42c54f80.chunk.js │ │ ├── 3685.39209b63.chunk.js │ │ ├── 4004.05e297fd.chunk.js │ │ ├── 4021.e04e7e55.chunk.js │ │ ├── 4482.6da0ecb0.chunk.js │ │ ├── 4494.b35991fe.chunk.js │ │ ├── 4681.fc821bbf.chunk.js │ │ ├── 4694.ead63044.chunk.js │ │ ├── 4721.108179ed.chunk.js │ │ ├── 4724.444c1588.chunk.js │ │ ├── 4785.63597083.chunk.js │ │ ├── 4831.595555ab.chunk.js │ │ ├── 4994.37361da4.chunk.js │ │ ├── 5299.478bd9da.chunk.js │ │ ├── 5599.bc0ea1af.chunk.js │ │ ├── 5783.1caefe8d.chunk.js │ │ ├── 6093.31927157.chunk.js │ │ ├── 6134.35028dd4.chunk.js │ │ ├── 6449.98df60ee.chunk.js │ │ ├── 6679.e1debb86.chunk.js │ │ ├── 6679.e1debb86.chunk.js.LICENSE.txt │ │ ├── 6802.615748ec.chunk.js │ │ ├── 6865.93ac0107.chunk.js │ │ ├── 698.096ccf39.chunk.js │ │ ├── 7125.70a408b8.chunk.js │ │ ├── 7135.0b3044f9.chunk.js │ │ ├── 7579.ab4d23ff.chunk.js │ │ ├── 7696.f57124bd.chunk.js │ │ ├── 8050.8be50b25.chunk.js │ │ ├── 8097.99be3737.chunk.js │ │ ├── 8097.99be3737.chunk.js.LICENSE.txt │ │ ├── 8186.18b3fb43.chunk.js │ │ ├── 8186.18b3fb43.chunk.js.LICENSE.txt │ │ ├── 865.6b1ec0ad.chunk.js │ │ ├── 8801.d4ef7a00.chunk.js │ │ ├── 8850.2182e134.chunk.js │ │ ├── 8957.b697f7a2.chunk.js │ │ ├── 9001.5422e085.chunk.js │ │ ├── 9005.baeb8ab0.chunk.js │ │ ├── 9166.e335c84b.chunk.js │ │ ├── 9166.e335c84b.chunk.js.LICENSE.txt │ │ ├── 9311.26e5e737.chunk.js │ │ ├── 9320.3ed3fc02.chunk.js │ │ ├── 9449.186aa58b.chunk.js │ │ ├── 9449.186aa58b.chunk.js.LICENSE.txt │ │ ├── 9813.ff50cf3b.chunk.js │ │ ├── main.b686df03.js │ │ └── main.b686df03.js.LICENSE.txt │ └── media │ ├── KaTeX_AMS-Regular.73ea273a72f4aca30ca5.woff2 │ ├── KaTeX_AMS-Regular.853be92419a6c3766b9a.ttf │ ├── KaTeX_AMS-Regular.d562e886c52f12660a41.woff │ ├── KaTeX_Caligraphic-Bold.7489a2fbfb9bfe704420.ttf │ ├── KaTeX_Caligraphic-Bold.a1abf90dfd72792a577a.woff2 │ ├── KaTeX_Caligraphic-Bold.d757c535a2e5902f1325.woff │ ├── KaTeX_Caligraphic-Regular.7e873d3833eb108a0758.ttf │ ├── KaTeX_Caligraphic-Regular.d6484fce1ef428d5bd94.woff2 │ ├── KaTeX_Caligraphic-Regular.db074fa22cf224af93d7.woff │ ├── KaTeX_Fraktur-Bold.354501bac435c3264834.woff │ ├── KaTeX_Fraktur-Bold.4c761b3711973ab04edf.ttf │ ├── KaTeX_Fraktur-Bold.931d67ea207ab37ee693.woff2 │ ├── KaTeX_Fraktur-Regular.172d3529b26f8cedef6b.woff2 │ ├── KaTeX_Fraktur-Regular.6fdf0ac577be0ba82a4c.woff │ ├── KaTeX_Fraktur-Regular.ed305b5434865e06ffde.ttf │ ├── KaTeX_Main-Bold.0c3b8929d377c0e9b2f3.woff │ ├── KaTeX_Main-Bold.39890742bc957b368704.woff2 │ ├── KaTeX_Main-Bold.8169508bf58f8bd92ad8.ttf │ ├── KaTeX_Main-BoldItalic.20f389c4120be058d80a.woff2 │ ├── KaTeX_Main-BoldItalic.428978dc7837d46de091.woff │ ├── KaTeX_Main-BoldItalic.828abcb200061cffbaae.ttf │ ├── KaTeX_Main-Italic.fa675e5e4bec9eb250b6.ttf │ ├── KaTeX_Main-Italic.fd947498bc16392e76c2.woff │ ├── KaTeX_Main-Italic.fe2176f79edaa716e621.woff2 │ ├── KaTeX_Main-Regular.4f35fbcc9ee8614c2bcc.woff │ ├── KaTeX_Main-Regular.9eba1d77abcf2aa6e94e.ttf │ ├── KaTeX_Main-Regular.f650f111a3b890d116f1.woff2 │ ├── KaTeX_Math-BoldItalic.3f07ed67f06c720120ce.woff │ ├── KaTeX_Math-BoldItalic.bf2d440b3a42ea78a998.ttf │ ├── KaTeX_Math-BoldItalic.dcbcbd93bac0470b462d.woff2 │ ├── KaTeX_Math-Italic.6d3d25f4820d0da8f01f.woff2 │ ├── KaTeX_Math-Italic.8a5f936332e8028c7278.ttf │ ├── KaTeX_Math-Italic.96759856b4e70f3a8338.woff │ ├── KaTeX_SansSerif-Bold.5b49f4993ae22d7975b4.ttf │ ├── KaTeX_SansSerif-Bold.95591a929f0d32aa282a.woff2 │ ├── KaTeX_SansSerif-Bold.b9cd458ac6d5889ff9c3.woff │ ├── KaTeX_SansSerif-Italic.7d393d382f3e7fb1c637.woff2 │ ├── KaTeX_SansSerif-Italic.8d593cfaa96238d5e2f8.woff │ ├── KaTeX_SansSerif-Italic.b257a18c016f37ee4543.ttf │ ├── KaTeX_SansSerif-Regular.02271ec5cb9f5b4588ac.woff │ ├── KaTeX_SansSerif-Regular.2f7bc363fc5424ebda59.ttf │ ├── KaTeX_SansSerif-Regular.cd5e231e0cc53b2cb2c0.woff2 │ ├── KaTeX_Script-Regular.073b3402d036714b4370.woff │ ├── KaTeX_Script-Regular.c81d1b2a4b75d3eded60.woff2 │ ├── KaTeX_Script-Regular.fc9ba5249878cd8f8d88.ttf │ ├── KaTeX_Size1-Regular.0108e89c9003e8c14ea3.woff │ ├── KaTeX_Size1-Regular.6de7d4b539221a49e9e2.ttf │ ├── KaTeX_Size1-Regular.6eec866c69313624be60.woff2 │ ├── KaTeX_Size2-Regular.2960900c4f271311eb36.woff2 │ ├── KaTeX_Size2-Regular.3a99e70aee4076660d38.woff │ ├── KaTeX_Size2-Regular.57f5c1837853986ea1db.ttf │ ├── KaTeX_Size3-Regular.7947224e8a9914fa332b.woff │ ├── KaTeX_Size3-Regular.8d6b6822586eea3d3b20.ttf │ ├── KaTeX_Size3-Regular.e1951519f6f0596f7356.woff2 │ ├── KaTeX_Size4-Regular.4ad7c7e8bb8d10a34bb7.ttf │ ├── KaTeX_Size4-Regular.aeffd8025cba3647f1a6.woff │ ├── KaTeX_Size4-Regular.e418bf257af1052628d8.woff2 │ ├── KaTeX_Typewriter-Regular.4c6b94fd1d07f8beff7c.woff │ ├── KaTeX_Typewriter-Regular.c295e7f71970f03c0549.woff2 │ ├── KaTeX_Typewriter-Regular.c5c02d763c89380dcb4e.ttf │ ├── SourceCodePro-Bold.17a68a0751a813474a0e.woff2 │ ├── SourceCodePro-BoldItalic.d45b7a3df103d441d78b.woff2 │ ├── SourceCodePro-Italic.ec122a420df4175e74f2.woff2 │ ├── SourceCodePro-Regular.84b900b88d09398d86b0.woff2 │ ├── SourceCodePro-SemiBold.17291d1c493cb25eb2c3.woff2 │ ├── SourceCodePro-SemiBoldItalic.f879ae27307c3926522b.woff2 │ ├── SourceSansPro-Bold.118dea98980e20a81ced.woff2 │ ├── SourceSansPro-BoldItalic.1d664be59d2eb5fef029.woff2 │ ├── SourceSansPro-Italic.8a9bfea74d43927d6eec.woff2 │ ├── SourceSansPro-Regular.0d69e5ff5e92ac64a0c9.woff2 │ ├── SourceSansPro-SemiBold.abed79cd0df1827e18cf.woff2 │ ├── SourceSansPro-SemiBoldItalic.befb0a2824eabc5ce36b.woff2 │ ├── SourceSerifPro-Bold.d7975b56594770699ae0.woff2 │ ├── SourceSerifPro-BoldItalic.913697a7178b128caa4c.woff2 │ ├── SourceSerifPro-Italic.d3529cb5797663ac5d88.woff2 │ ├── SourceSerifPro-Regular.e6c37aa3926474cc93e1.woff2 │ ├── SourceSerifPro-SemiBold.5c1d378dd5990ef334ca.woff2 │ ├── SourceSerifPro-SemiBoldItalic.249e948b885d0c7d30a8.woff2 │ ├── balloon-0.0e6e96ece0ea0719d6b6.png │ ├── balloon-1.a503c271109e9a0fdd2d.png │ ├── balloon-2.872250f3bf3a8965d655.png │ ├── balloon-3.e772fafd3bfc55fb439b.png │ ├── balloon-4.1a74162473d81c9cb9ee.png │ ├── balloon-5.edb75c74c54165b29856.png │ ├── checkmark.29851c8e9e6ef0c3d6c1e4efe3c1bb9e.svg │ ├── fireworks.0906f02ea43f1018a6d2.gif │ ├── flake-0.beded754e8024c73d9d2.png │ ├── flake-1.8077dc154e0bf900aa73.png │ ├── flake-2.e3f07d06933dd0e84c24.png │ ├── icon_running.de1119daebe465fa5263.gif │ ├── logo.83ae4f2fb87e38be7cbb8a5d2beb64d2.svg │ └── rocket.b75b17d2b0a063c6cea230d1a9d77f1e.svg ├── stlite-manifest.json └── streamlit_app └── streamlit_app.py

The package.json

{
  "name": "TestApp",
  "version": "0.1.0",
  "main": "./build/electron/main.js",
  "scripts": {
    "dump": "dump-stlite-desktop-artifacts",
    "serve": "cross-env NODE_ENV=production electron .",
    "pack": "electron-builder --dir",
    "dist": "electron-builder",
    "postinstall": "electron-builder install-app-deps"
  },
  "build": {
    "files": ["build/**/*"],
    "directories": {
      "buildResources": "assets"
    }
  },
  "devDependencies": {
    "@stlite/desktop": "^0.51.0",
    "cross-env": "^7.0.3",
    "electron": "^28.2.1",
    "electron-builder": "^24.9.1"
  },
  "stlite": {
    "desktop": {
      "nodeJsWorker": true,
      "nodefsMountpoints": {
        "/mnt": "/Users/Shared/Appfiles"
      }
    }
  }
}

Yes build/pyodide/protobuf-4.23.1-cp311-cp311-emscripten_3_1_45_wasm32.whl does exist

Apr 09 '24 12:04 drankush

Hey! @whitphx what do you think of this.

Apr 26 '24 17:04 drankush

Thank you, I could reproduce the error. Looks like this happens if the NodeJS worker mode is enabled and at least one of st_files_connection, PyYAML, and streamlit-tags is installed. Let me take more to investigate this.

Apr 28 '24 08:04 whitphx

A quick workaround is to add "asar": false to the "build" field in your package.json and run yarn dist again, while it's not recommended in general.

As a dev note, the cause of this error is Pyodide's loadPackage() tries to write its package directory in NodeJS mode while the target dir is read-only in the case of the packaged Electron app as it's an ASAR archive. Looks like this has to be fixed in Pyodide. Will work on it.

Apr 29 '24 01:04 whitphx

Resolved by #934. The patch is included in the 0.56.0 release.

May 29 '24 11:05 whitphx