python
Provide a way to deal with obsolete loop references in the stdlib objects
Things like Queue have self._loop reference. If you change the loop, suddenly you have weird errors occuring and you need to hunt down every single object having a reference to the previous loop, and either update it to the new one, or change your bootstrap code so that they are constructed after the loop change.
This mean you need to read a lot of asyncio's code when this happen just to be able to find all those references. This is particularly a problem if somebody else code is changing the loop, in that case you have twice the work : finding the references and where they are created relatively to that code.
I don't have a solution to this problem, I'm just opening the ticket so we can debate on how to deal with this issue, as giving access to the loop mean people will change it, and the stlib is very static in the way it deals with the loop.
Some ideas:
- implement #382 and make sure the stdlib uses it to display warnings about obsolete references;
- implement some kind of loop reference registry;
- just declare "be careful about it, it's your problem and your job". At least we'll know where we stand.
- create a setting, which, when set, will raise an exception if you try to create several loops in the same context, so you can easily debug it;
Hi,
It seems very brittle to change the loop to which an object is tied after its creation, as it is almost impossible to keep track of the asynchronous actions scheduled by this object. In the case of Queue, what would happen if self._loop.call_soon(self._do_something) is called and the loop changes right before the callback is executed? It's even worse If the object performs IO, as I can imagine many ways to miss an event notification.
Currently, the stdlib raises an exception if one is invoking a closed loop.
It seems to me that it is up to the developer to implement the desired policy, with a registry (for instance a loop per thread), or raise an exception in new_event_loop() if a default event loop is set.
Do you know a case where it make sense to use several loops in the same thread? If one is running, the other loop will not be able to run concurrently.
just declare "be careful about it, it's your problem and your job". At least we'll know where we stand.
This seems to be the most preferable option. It's impossible to protect people from all edge cases they can encounter when they use multiple loops within one process – best thing is to explain they never should do that.
@Martiusweb: I encountered the problem because some pytest plugin was recreating a loop, and it took a while, digging into asyncio source code to find out Queue was olding a reference to the old one. I have been programming in Python for many years, and yet it was tedious to debug.
Now picture most dev. They are already having a hard time understanding how to use asyncio right now (see europython...). Imagine they encounter that. It will be a nightmare for them to debug.
It would be great to provide something to ease the pain. Granted, I don't know what "something" is.
What I know is that I now have been programming with asyncio for several months, and there is no way most average dev will have the patience or energy to do anything complicated with it if we don't make their life easier. The explicit event loop and the open door policy of Python make it too easy to scramble things, and the complex nature of asyncio makes it very hard to find the puzzle pieces.
@1st1: well, at least it would be nice to help a lost dev to pinpoint the problem. It's a hard thing to do, I agree. Again, I don't have a clean solution in mind. Currently my asyncio project is riddle with catching exceptions and raising a more explicit one to ease debugging. It's the first time I have to do that in Python.