anyio
anyio copied to clipboard
agronholm
Thread race condition in _eventloop.get_asynclib() (with fix)
I have a threading Web spider using httpx that is triggering a race condition in anyio/_core/_eventloop.py's get_asynclib() function. This race can be triggered if you have a lot of threads trying to initialize their own asyncio event loop and initializing httpx all at once. It happens intermittently -- sometimes we don't trigger the race, sometimes we do. When we do, the threads catch get_asynclib() mid-initialization and certain attributes that should be present are not available. I am using anyio 3.5.0, httpx 0.22.0 and python3.7.
An ugly but functioning patch which I can confirm resolves the issue is below:
get_asynclib_thread_fix.patch.txt
In my test patch, I made the lock cover essentially the entire get_asynclib() call -- it could potentially be made tighter, with questionable benefit -- not sure.
I'm very hesitant to add locking to get_asynclib() which is "hot" code. I'll take a look at this and see if I can find a better solution.
I do question the use case though – why would you run multiple event loops?
If you are using threads, each thread needs its own event loop. So if you ever use ThreadPoolExecutor, and you want to use async inside, each one needs its own event loop. That is the use case in my code. In my case, each thread is performing network operations so to use httpx, I require an event loop to exist for the lifetime of the thread.
I wonder if this problem would still occur if you were to use ProcessPoolExecutor with 'spawn' context, rather than threads. That way each event loop would be running in a separate memory space and your event loops would be isolated. Therefore, there would be no chance of threads racing against each other due to shared state. I think separate processes is the correct solution here in any case, to give you true parallelism.
