micropip Dependency errors while trying to install `rst-to-myst`

🐛 Bug

hi folks! first time Pyodide user here. I'm trying to install rst-to-myst on the Pyodide REPL and I'm getting errors like these:

>>> await micropip.install("rst-to-myst")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 548, in install
    await transaction.gather_requirements(requirements)...
ValueError: Requested 'mdit-py-plugins<0.4.0,>=0.3.0', but mdit-py-plugins==0.2.8 is already installed

it looks like micropip failed to solve the dependencies correctly. if I try to amend the situation, I reach a dead end:

>>> await micropip.install("mdit-py-plugins<0.4.0")
>>> micropip.list()
Name            | Version | Source
--------------- | ------- | -------
packaging       | 21.3    | pyodide
micropip        | 0.1     | pyodide
pyparsing       | 3.0.9   | pyodide
mdurl           | 0.1.2   | pypi
markdown-it-py  | 2.1.0   | pypi
mdit-py-plugins | 0.3.0   | pypi
distutils       | 1.0.0   | pyodide
>>> await micropip.install("rst-to-myst")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 548, in install
    await transaction.gather_requirements(requirements)
...
    raise ValueError(
ValueError: Requested 'markdown-it-py~=1.0', but markdown-it-py==2.1.0 is already installed
>>> await micropip.install("markdown-it-py~=1.0")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 548, in install
    await transaction.gather_requirements(requirements)...
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 336, in check_version_satisfied
    raise ValueError(
ValueError: Requested 'markdown-it-py~=1.0', but markdown-it-py==2.1.0 is already installed

There's probably a problem with the dependency resolution here, because using pip locally or pip-tools --resolver=backtracing works. I managed to pin down the versions that work for me, after struggling with ruamel.yaml (see pyodide/pyodide#3034).

To Reproduce

>>> await micropip.install("rst-to-myst")

Expected behavior

I guess I have sort of an unrealistic expectation here: that micropip solves the dependencies in exactly the same way as pip. But even if that's not the case and there are suitable differences, I should be able to manually upgrade a package after it has been installed?

Environment

Pyodide Version: How can I know? I'm on https://pyodide.org/en/latest/console.html
Browser version: Firefox 104.0

Aug 29 '22 17:08 astrojuanlu

I guess I have sort of an unrealistic expectation here: that micropip solves the dependencies in exactly the same way as pip. But even if that's not the case and there are suitable differences, I should be able to manually upgrade a package after it has been installed?

Yes, micropip does dependency resolution in a sort of ad-hoc way as it is an NP-hard problem. It would be nice if we also have a backtracking resolver, but we simply don't have enough people and time to implement it (anyone interested is welcome :) ).

The current workaround is passing deps=False in micropip.install to disable dependency resolution and manually install dependencies one-by-one.

Aug 30 '22 01:08 ryanking13

Well I think another issue is that we don't really want to do tons of downloads. I lean against implementing a backtracking resolver to use in the browser. If the naive algorithm doesn't work, it would be nice to be able to resolve ahead of time with a better out-of-browser resolver and then create a lock file from that. Otherwise, the install time could be very long.

On the other hand, there are two real problems:

micropip is nondeterministic
micropip has no logging and no good way to say what went wrong in the dependency resolution

Aug 30 '22 01:08 hoodmane

Is this essentially the same problem?

This works:

Welcome to the Pyodide 0.27.4 terminal emulator 🐍
Python 3.12.7 (main, Mar 18 2025 13:27:55) on WebAssembly/Emscripten

>>> import micropip; await micropip.install(['opentelemetry-sdk==1.30.0'], verbose=True)

Collecting opentelemetry-sdk==1.30.0
  Downloading opentelemetry_sdk-1.30.0-py3-none-any.whl
Collecting opentelemetry-semantic-conventions==0.51b0
  Downloading opentelemetry_semantic_conventions-0.51b0-py3-none-any.whl
Collecting opentelemetry-api==1.30.0
  Downloading opentelemetry_api-1.30.0-py3-none-any.whl
Requirement already satisfied: opentelemetry-api==1.30.0 (1.30.0)
Collecting deprecated>=1.2.6
  Downloading Deprecated-1.2.18-py2.py3-none-any.whl
Collecting importlib-metadata<=8.5.0,>=6.0
  Downloading importlib_metadata-8.5.0-py3-none-any.whl
Requirement already satisfied: deprecated>=1.2.6 (1.2.18)
Collecting deprecated>=1.2.6
  Downloading Deprecated-1.2.18-py2.py3-none-any.whl
Collecting zipp>=3.20
  Downloading zipp-3.21.0-py3-none-any.whl
Installing collected packages: typing-extensions, wrapt, wrapt, deprecated, deprecated, opentelemetry-semantic-conventions, zipp, importlib-metadat
a, opentelemetry-api, opentelemetry-sdk
Successfully installed typing-extensions-4.11.0, wrapt-1.16.0, wrapt-1.16.0, deprecated-1.2.18, deprecated-1.2.18, opentelemetry-semantic-conventio
ns-0.51b0, zipp-3.21.0, importlib-metadata-8.5.0, opentelemetry-api-1.30.0, opentelemetry-sdk-1.30.0

>>> import micropip; await micropip.install(['opentelemetry-instrumentation==0.51b0'], verbose=True)

Collecting opentelemetry-instrumentation==0.51b0
  Downloading opentelemetry_instrumentation-0.51b0-py3-none-any.whl
Requirement already satisfied: opentelemetry-semantic-conventions==0.51b0 (0.51b0)
Requirement already satisfied: packaging>=18.0 (24.2)
Requirement already satisfied: wrapt<2.0.0,>=1.0.0 (1.16.0)
Requirement already satisfied: opentelemetry-api~=1.4 (1.30.0)
Installing collected packages: opentelemetry-instrumentation
Successfully installed opentelemetry-instrumentation-0.51b0

But this fails:

Welcome to the Pyodide 0.27.4 terminal emulator 🐍
Python 3.12.7 (main, Mar 18 2025 13:27:55) on WebAssembly/Emscripten

>>> import micropip; await micropip.install(['opentelemetry-sdk==1.30.0', 'opentelemetry-instrumentation==0.51b0'], verbose=True)

Collecting opentelemetry-sdk==1.30.0
  Downloading opentelemetry_sdk-1.30.0-py3-none-any.whl
Collecting opentelemetry-instrumentation==0.51b0
  Downloading opentelemetry_instrumentation-0.51b0-py3-none-any.whl
Requirement already satisfied: packaging>=18.0 (24.2)
Collecting opentelemetry-api==1.30.0
  Downloading opentelemetry_api-1.30.0-py3-none-any.whl
Requirement already satisfied: opentelemetry-api~=1.4 (1.30.0)
Collecting opentelemetry-api~=1.4
  Downloading opentelemetry_api-1.31.0-py3-none-any.whl
Collecting opentelemetry-semantic-conventions==0.51b0
  Downloading opentelemetry_semantic_conventions-0.51b0-py3-none-any.whl
Requirement already satisfied: opentelemetry-semantic-conventions==0.51b0 (0.51b0)
Collecting opentelemetry-semantic-conventions==0.51b0
  Downloading opentelemetry_semantic_conventions-0.51b0-py3-none-any.whl

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.12/site-packages/micropip/package_manager.py", line 133, in install
    return await install(
           ^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/install.py", line 53, in install
    await transaction.gather_requirements(requirements)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 55, in gather_requirements
    await asyncio.gather(*requirement_promises)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 62, in add_requirement
    return await self.add_requirement_inner(Requirement(req))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 152, in add_requirement_inner
    await self._add_requirement_from_package_index(req)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 214, in _add_requirement_from_package_index
    await self.add_wheel(wheel, req.extras, specifier=str(req.specifier))
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 262, in add_wheel
    await asyncio.gather(
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 55, in gather_requirements
    await asyncio.gather(*requirement_promises)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 59, in add_requirement
    return await self.add_requirement_inner(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 152, in add_requirement_inner
    await self._add_requirement_from_package_index(req)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 214, in _add_requirement_from_package_index
    await self.add_wheel(wheel, req.extras, specifier=str(req.specifier))
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 262, in add_wheel
    await asyncio.gather(
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 55, in gather_requirements
    await asyncio.gather(*requirement_promises)
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 59, in add_requirement
    return await self.add_requirement_inner(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 141, in add_requirement_inner
    satisfied, ver = self.check_version_satisfied(req)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/micropip/transaction.py", line 86, in check_version_satisfied
    raise ValueError(
ValueError: Requested 'opentelemetry-api==1.30.0', but opentelemetry-api==1.31.0 is already installed

Mar 19 '25 13:03 alexmojaki

@alexmojaki

Is this essentially the same problem?

Not actually I guess, but it is something we need to fix too.

Mar 20 '25 08:03 ryanking13