`nest_asyncio` is incompatible with `uvloop`
nest_asyncio is used in the JSON-LD document loader methods (aries_cloudagent/vc/ld_proofs/document_loader.py)
The incompatibility was observed when working with the redis_events plugin in hyperledger/aries-acapy-plugins, which uses uvicorn. After attempting to update uvicorn from 0.16 to 0.29 over there, it appears that one of the changes introduces uvloop, causing an incompatibility with the nest_asyncio framework.
The error stacktrace is as follows:
2024-05-09 14:48:58,232 aries_cloudagent.core.dispatcher ERROR Handler error: validation_middleware
Traceback (most recent call last):
File "/home/aries/.local/lib/python3.9/site-packages/pyld/context_resolver.py", line 143, in _fetch_context
remote_doc = jsonld.load_document(url,
File "/home/aries/.local/lib/python3.9/site-packages/pyld/jsonld.py", line 6583, in load_document
=================
remote_doc = options['documentLoader'](url, options)
File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/vc/ld_proofs/document_loader.py", line 111, in __call__
document = loop.run_until_complete(coroutine)
File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 518, in uvloop.loop.Loop._run
RuntimeError: this event loop is already running.
Looking at the commit history, I can see that nest_asyncio was introduced as a workaround:
https://github.com/hyperledger/aries-cloudagent-python/commit/40e40eb561fc50c0bdbcdb41fdeda7b0bd3ac1e4
fix: Replace hack within document loader
After talking with Timo, the document loader uses a seperate thread due
to the fact that PyLD is synchronous code and calls into the document
loader (which requires asynchronous code). When swapping out the cache
with a Redis based cache, new exceptions rose up due to this
implementation. No matter what I tried, it seemed impossible to keep the
separate thread/asyncio event loop due to the error `got Future attached
to a different loop`.
Switching to `nest_asyncio` over using a separate thread/event loop
resolves the issues that I was observing.
Signed-off-by: Colton Wolkins (Indicio work address) <[email protected]>
I'm logging this as an issue so that we can find an alternative to nest_asyncio, which is compatible with uvloop.
A refactoring of asyncio event loops may in any case be desirable because "uvloop makes asyncio 2-4x faster": https://github.com/MagicStack/uvloop