pystorm
pystorm copied to clipboard
pystorm
[WIP] Add AsyncComponent, AsyncBolt, and AsyncSpout classes.
There are no tests for these yet, but these are the updated versions of what @kbourgoin wrote long ago.
We also monkey patch things with gevent if it's available, since that makes the AsyncComponent subclasses faster.
Adding tests for this when I get a chance. So far I found a few broken things that I've fixed. I'm trying to write the code so that it'll just use threading if gevent isn't available and then rely on monkey patching to make it uses greenlets if available. I'm not sure if this is the best approach, as I've actually never tried to use gevent before.
This isn't quite working yet, so maybe an asyncio-based approach as discussed in #8 makes more sense than the threading approach I was taking.
What is needed to port Component to asyncio? I was digging in it and it seems to me that only run() loop while True: needs to be taken care of. Or is there some other malicious things that I didn't find? Is there some kind of skeleton I can stick with? Since loading information from external web services is terribly slow when not using threads and keep Bolt in "busy waiting" state.
@Darkless012, if you mean in general, then we also need to make sure that we're reading and writing to stdin/stdout in an async fashion. That's the main benefit of making this all async: you don't have blocking IO.
This particular PR I haven't thought about it a long time, but I actually think I'll be picking this back up again soon for some stuff we're working on at Parse.ly.
Ugh. I started poking this again, and I don't think we'll be able to use asyncio for a little while for this, because I have not been able to find anyway to do async reading from stdin/stdout. There's aiofiles, but it doesn't work with stdin/stdout (Tinche/aiofiles#8). Although I guess we could leave the stdin/stdout reading bit as blocking, and just get rid of our while True bits and try to use the aio event loop.
After giving up on trying to get this switched over to asyncio, I tried again to get this PR actually working. I found a couple potential testing issues (like us not stopping all the threads before moving on to the next test), so I think I am close to actually having this working.