aio-libs
Multiple event loops
The code currently seems to assume it will never be accessed by multiple event loops at the same time.
I have made a minimal reproduction of the issue.
Yes, it is a known limitation. It is mentioned in README, third point.
I do not think the problem of multiple loops will ever be solved in Janus, as it is an architectural limitation. Janus relies on existing asyncio primitives that bind to the current event loop when used (when created on Python 3.10 and below), and thus does not support multiple event loops - this is directly related to the known non-thread-safety of asyncio primitives. Solving this issue would require significant changes to the source code, which has not changed architecturally since the first releases, that is, about 10 years (you can see a list of all current source code changes via GitHub).
For the case of multiple event loops, you can use Culsans.
If the issue is not that Janus should support multiple event loops, but that it should raise an exception in such a case, I do not see the need for it. RuntimeError is already raised by asyncio primitives. And by Janus itself.
Note that there are some bugs with binding in the case of multiple event loops that my one PR is fixing, but it still has not merged.
I have made a minimal reproduction of the issue.
By the way, you have the wrong expected order in the third test. "worker 2: <timeout>" will be printed before "worker 3: 69", because there is 0.5 seconds difference between them. If you swap them and replace import janus with import culsans as janus, the tests will pass. But this may not be guaranteed because of the overhead of asyncio.run() - it may be better to synchronize threads with a barrier at start (though, using real time in tests is always a pain; I think tests should be more deterministic, with no timeouts).
I do not think the problem of multiple loops will ever be solved in Janus, as it is an architectural limitation. Janus relies on existing asyncio primitives that bind to the current event loop when used (when created on Python 3.10 and below)
So, just to confirm, a solution which involves building a threaded Lock object for internal use that can be acquired both by sync and async code, would be out of the question?
So, just to confirm, a solution which involves building a threaded Lock object for internal use that can be acquired both by sync and async code, would be out of the question?
Yes, because the cause of the issue is that Janus uses asyncio.Condition for the asynchronous API.
Actually, the issue has been raised indirectly before. My Culsans was the answer. It uses aiologic.Condition, which is both async-aware and thread-aware. But uses a sync-only lock to avoid deadlocks when mixing synchronous and asynchronous calls, which is in standard tests.
The primitive you described is also available in aiologic. And it is also my library.
In general, I will say the following. The Janus architecture is a symbiosis of queue.Queue and asyncio.Queue, almost copy-paste. It simply takes the standard implementations and binds them together using event loop methods. Janus also includes modified standard tests (test_sync.py and test_async.py) that borrow from python/cpython but ignore the license terms (PSF-2.0).
Culsans is a separate library that provides Janus-like queues, but solves problems that Janus does not. Instead of using different condition variables (threading.Condition and asyncio.Condition instances) for each API, it uses common ones - aiologic.Condition instances. It is careful about licensing conditions and uses its own implementation, but using queue.Queue coding patterns for reliability and easier understanding.
Dependabot does almost all the commits in this repository. I do not think Janus will evolve, and with the introduction of the "Sponsor" button, I started to doubt even more strongly that new PRs will ever be accepted. Like I said, there is Culsans, but... It is hard to get through titans, especially in these times of AI hype.