aries-cloudagent-python icon indicating copy to clipboard operation
aries-cloudagent-python copied to clipboard

`nest_asyncio` is incompatible with `uvloop`

Open ff137 opened this issue 1 year ago • 0 comments

nest_asyncio is used in the JSON-LD document loader methods (aries_cloudagent/vc/ld_proofs/document_loader.py)

The incompatibility was observed when working with the redis_events plugin in hyperledger/aries-acapy-plugins, which uses uvicorn. After attempting to update uvicorn from 0.16 to 0.29 over there, it appears that one of the changes introduces uvloop, causing an incompatibility with the nest_asyncio framework. The error stacktrace is as follows:

2024-05-09 14:48:58,232 aries_cloudagent.core.dispatcher ERROR Handler error: validation_middleware
Traceback (most recent call last):
  File "/home/aries/.local/lib/python3.9/site-packages/pyld/context_resolver.py", line 143, in _fetch_context
    remote_doc = jsonld.load_document(url,
  File "/home/aries/.local/lib/python3.9/site-packages/pyld/jsonld.py", line 6583, in load_document

=================
    remote_doc = options['documentLoader'](url, options)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/vc/ld_proofs/document_loader.py", line 111, in __call__
    document = loop.run_until_complete(coroutine)
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 518, in uvloop.loop.Loop._run
RuntimeError: this event loop is already running.

Looking at the commit history, I can see that nest_asyncio was introduced as a workaround: https://github.com/hyperledger/aries-cloudagent-python/commit/40e40eb561fc50c0bdbcdb41fdeda7b0bd3ac1e4

fix: Replace hack within document loader

After talking with Timo, the document loader uses a seperate thread due
to the fact that PyLD is synchronous code and calls into the document
loader (which requires asynchronous code). When swapping out the cache
with a Redis based cache, new exceptions rose up due to this
implementation. No matter what I tried, it seemed impossible to keep the
separate thread/asyncio event loop due to the error `got Future attached
to a different loop`.

Switching to `nest_asyncio` over using a separate thread/event loop
resolves the issues that I was observing.

Signed-off-by: Colton Wolkins (Indicio work address) <[email protected]>

I'm logging this as an issue so that we can find an alternative to nest_asyncio, which is compatible with uvloop.

A refactoring of asyncio event loops may in any case be desirable because "uvloop makes asyncio 2-4x faster": https://github.com/MagicStack/uvloop

ff137 avatar May 09 '24 19:05 ff137