Make aiohttp work in Jupyterlite

[MarcSkovMadsen]

Is your feature request related to a problem?

I'm a contributor to Panel which can run on many platforms including Panelite (~Jupyterlite). We would like to be able to use Jupyterlite to provide an easy to use platform to showcase our framework and help our users learn how to use it.

I believe this is a general trend in Python that frameworks and users start moving to the browser powered by Pyodide, PyScript and Jupyterlite for education and for lighter data analytics.

One of Panels reference notebooks shows how to use Panel with async and aiohttp. The problem is that I cannot import aiohttp with piplite.

You can reproduce as described below

• Open Jupyterlite https://jupyterlite.readthedocs.io/en/latest/_static/lab/index.html
• Open a Pyodide notebook
• Run the code below

import piplite
await piplite.install(['aiohttp'])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 2
      1 import piplite
----> 2 await piplite.install(['aiohttp'])

File /lib/python3.11/site-packages/piplite/piplite.py:103, in _install(requirements, keep_going, deps, credentials, pre)
    101 """Invoke micropip.install with a patch to get data from local indexes"""
    102 with patch("micropip._micropip._get_pypi_json", _get_pypi_json):
--> 103     return await _micropip.install(
    104         requirements=requirements,
    105         keep_going=keep_going,
    106         deps=deps,
    107         credentials=credentials,
    108         pre=pre,
    109     )

File /lib/python3.11/site-packages/micropip/_micropip.py:576, in install(requirements, keep_going, deps, credentials, pre)
    566 wheel_base = Path(getsitepackages()[0])
    568 transaction = Transaction(
    569     ctx=ctx,
    570     ctx_extras=[],
   (...)
    574     fetch_kwargs=fetch_kwargs,
    575 )
--> 576 await transaction.gather_requirements(requirements)
    578 if transaction.failed:
    579     failed_requirements = ", ".join([f"'{req}'" for req in transaction.failed])

File /lib/python3.11/site-packages/micropip/_micropip.py:342, in Transaction.gather_requirements(self, requirements)
    339 for requirement in requirements:
    340     requirement_promises.append(self.add_requirement(requirement))
--> 342 await gather(*requirement_promises)

File /lib/python3.11/site-packages/micropip/_micropip.py:349, in Transaction.add_requirement(self, req)
    346     return await self.add_requirement_inner(req)
    348 if not urlparse(req).path.endswith(".whl"):
--> 349     return await self.add_requirement_inner(Requirement(req))
    351 # custom download location
    352 wheel = WheelInfo.from_url(req)

File /lib/python3.11/site-packages/micropip/_micropip.py:457, in Transaction.add_requirement_inner(self, req)
    452 if self.check_version_satisfied(req):
    453     # Maybe while we were downloading pypi_json some other branch
    454     # installed the wheel?
    455     return
--> 457 await self.add_wheel(wheel, req.extras)

File /lib/python3.11/site-packages/micropip/_micropip.py:472, in Transaction.add_wheel(self, wheel, extras)
    470 await wheel.download(self.fetch_kwargs)
    471 if self.deps:
--> 472     await self.gather_requirements(wheel.requires(extras))
    474 self.wheels.append(wheel)

File /lib/python3.11/site-packages/micropip/_micropip.py:342, in Transaction.gather_requirements(self, requirements)
    339 for requirement in requirements:
    340     requirement_promises.append(self.add_requirement(requirement))
--> 342 await gather(*requirement_promises)

File /lib/python3.11/site-packages/micropip/_micropip.py:346, in Transaction.add_requirement(self, req)
    344 async def add_requirement(self, req: str | Requirement) -> None:
    345     if isinstance(req, Requirement):
--> 346         return await self.add_requirement_inner(req)
    348     if not urlparse(req).path.endswith(".whl"):
    349         return await self.add_requirement_inner(Requirement(req))

File /lib/python3.11/site-packages/micropip/_micropip.py:444, in Transaction.add_requirement_inner(self, req)
    441 metadata = await _get_pypi_json(req.name, self.fetch_kwargs)
    443 try:
--> 444     wheel = find_wheel(metadata, req)
    445 except ValueError:
    446     self.failed.append(req)

File /lib/python3.11/site-packages/micropip/_micropip.py:312, in find_wheel(metadata, req)
    309     if best_wheel is not None:
    310         return wheel
--> 312 raise ValueError(
    313     f"Can't find a pure Python 3 wheel for '{req}'.\n"
    314     f"See: {FAQ_URLS['cant_find_wheel']}\n"
    315     "You can use `micropip.install(..., keep_going=True)`"
    316     "to get a list of all packages with missing wheels."
    317 )

ValueError: Can't find a pure Python 3 wheel for 'multidict<5.0,>=4.5'.
See: https://pyodide.org/en/stable/usage/faq.html#micropip-can-t-find-a-pure-python-wheel
You can use `micropip.install(..., keep_going=True)`to get a list of all packages with missing wheels.

Describe the solution you'd like

That I can import and use aiohttp in Pyodide and Jupyterlite.

Describe alternatives you've considered

Removing the aiohttp notebook from the Panelite example notebooks.

Related component

Client

Additional context

When the import problem has been solved there is probably a lot more to solve to get aiohttp working.

The code we are using is

async def get_img(index):
    async with aiohttp.ClientSession() as session:
        async with session.get(f"https://picsum.photos/800/300?image={index}") as resp:
            return await resp.read()

I can see that multidict can be imported by it self. Its the version pinned that is the import problem.

Code of Conduct

• [X] I agree to follow the aio-libs Code of Conduct

[Dreamsorcerer]

Pin in 3.9 is < 7, so if it's just multidict version, this'll be resolved then.

[MarcSkovMadsen]

Thx. My guess is that more is needed to make aiohttp work in Pyodide and Jupyterlite.

For example there is quite some work in pyodide-http for widely used http libraries to make them work in the browser.

[6r17]

I'm not really into the idea of modifying the inner workings of a robust library such as aiohttp so "it works" with another library. For all I know, it should not be aiohttp to change it's behavior.

[MarcSkovMadsen]

Pyodide/ webassembly is not another library. Its a new platform for Python. Similar to Windows, OSX and Linux 👍

The ask is not to make it work with Panel. The ask is to also support the browser as a platform vis Pyodide/ PyScript/ Jupyterlite/ webassembly.

I understand this is not a high priority right now.

[6r17]

My bad i wrongly understood the request. I'm just a user tough ;) It's much clearer however !

[Dreamsorcerer]

I understand this is not a high priority right now.

Generally, if you want a feature, you'll likely have to do the work to implement it. Once it's merged in, we'll try and keep it maintained in the future. I'll try and get a beta release of 3.9 out soon, and then you can retest installation.

[joemarshall]

@Dreamsorcerer I was considering doing a patch to this library in pyodide-http. Would you have an interest in support directly in aiohttp? Because of the nature of webasembly/emscripten, it would only be client, and would have some limitations due to only being able to do whatever the browser running it does, but looking at the API it doesn't seem so far removed from async http in webasembly. It would be handy for me to push it here rather than to have to monkeypatch it in pyodide-http.

[Dreamsorcerer]

It depends on what the changes involve really. Can you throw together a draft PR without putting much time into it? Then we can decide on whether it'll be worth finishing off.

[rth]

FIY @hoodmane wrote a draft monkeypatch in https://github.com/pyodide/pyodide/pull/4282#issuecomment-1796510844 and that PR woulds build a binary pyodide/wasm wheel for aiohttp, on which it could be applied.

[hoodmane]

@Dreamsorcerer currently the patch for Pyodide might be a bit invasive. The main problem is that we cannot meaningfully respect the user's choice of Connector. Mostly people seem to pass a TCPConnector but in my interpretation they are more after the arguments they pass to it which control various settings. These settings all need to be reimplemented but timeouts can work in close to the same way.

I think the best thing to do would be to add a Pyodide-specific FetchConnector which is backed by a fetch handler (a function which implements the JavaScript fetch API). The default behavior could be to use globalThis.fetch but you could specify an alternative handler. Then in Pyodide if someone asks for a TCPConnector we rearrange it to be backed by FetchConnector in some hopefully not too invasive way. If they ask for a subclass of BaseConnector that isn't a subclass of FetchConnector or TCPConnector we could just raise.

[hoodmane]

I will try to make a draft PR as a point of discussion, maybe later this week.

[hoodmane]

Okay opened #7803.