scrapy-redis
scrapy-redis copied to clipboard
Whether scrapy-redis can handle this situation - request urls have priority?
Hi,
Whether scrapy-redis can handle this situation - request urls have priority? Such as I need a request URL_1 to get the basic info and another request URL_2 to get the detail (URL_1 has to before URL_2)?
Thanks a lot.
It doesn't. You would need to implement a priority queue in redis as backend.
Thank you @rmax firstly.
This is really a problem in distributed system.