Raphaël Vinot

Reverting https://github.com/scrapy-plugins/scrapy-splash/pull/269/commits/4154782b16c6248a5a9b8fe3ed0ebb1cbc600a60 will do just that, so just go for it.

FYI, it is supported on AIL by this module: https://github.com/CIRCL/AIL-framework/blob/master/bin/Helper.py I will see if it is possible to implement it.

Protip: never use threading in python. Been here, done that, bad idea. My approach on AIL is to run processes poping values from a set and pushing the output on...

Here is my model for one queue in AIL  You can have multiple Queues subscribing to the same publisher.

With my solution, you can make loops, which is a feature and not a bug because I fetch onions and process them again :) Obviously, it only work if you...

The schema I did yesterday is for one queue. in my terminology, Alienvault-collector is one queue, alienvault-parser is one queue, cymru-to-asn is one queue. it doesn't make any sense to...

Here is the very initial version of the queuing code I'm porting to a standalone module from AIL: https://github.com/Rafiot/MultiProcQueue It is quite far from being ready, but most of the...

The code is almost ready, I use the same concepts as IntelMQ to make it an (easy enough) drop-in replacement. I will create a branch in IntelMQ for testing ASAP....

Short update: I'm going to update my code to use disque[1] instead of my custom queuing system: it seems much more reliable, is properly supported, and will much more easily...

mine is, I think, more reliable for now. But also not used in production either so I cannot really tell. I really want to give a try to disque, because...