streamz
streamz copied to clipboard
python-streamz
AnyIO implementation of asynchronous streams
Using streamz with an event loop such as trio is currently not possible since Tornado only supports AsyncIO if it is available.
If we had an implementation that uses AnyIO, we could use any event loop we'd desire including tornado.
Indeed, a dask client requires an event loop. It is possible to supply an event loop instance at client creation, and I don't know exactly what it required of that loop instance - probably anything asyncio compliant. However, dask's loop can also run in a separate thread, so maybe it is possible to have two loops going (which may get complicated when actually dealing with dask-streamz).
Have you tried using the anyIO (or other) loop for dask too? That would be the favourite solution if it works. Otherwise, I think I do basically agree that streamz should work regardless, but in practice, it may be tricky to demarcate dask-free parts of the code.
There is no AnyIO implementation for dask. There's an AsyncIO implementation and a Tornado implementation. I have never used dask so no on both accounts.
It might be possible to run dask with trio-asyncio but I haven't really checked.
I want to implement streamz on another framework.
This framework should not tie itself to a specific event loop. What if the next best thing comes around and everybody wants to use it?
True, but being able to use Dask is important too. If you can implement anyIO (or something else) for the non-Dask code without greatly complicating the code, we would consider it. Of course we would still prefer smooth interaction between streamz and Dask on any io-loop.
If you'd like to integrate Dask with other event loops I suggest taking it upstream.
I took a look at the code. It is currently impossible since you are tightly coupled with the Tornado event loop.
xref: https://github.com/python-trio/trio-asyncio/issues/22#issuecomment-586974857
Would uvloop, which is asyncio compatible, be a solution for you?
No. I need trio support.