scrapy-redis icon indicating copy to clipboard operation
scrapy-redis copied to clipboard

Published 20 hours ago • verified publisher icon rmax

Whether scrapy-redis can handle this situation - request urls have priority?

Open sunnypandas opened this issue 7 years ago • 2 comments

Hi,

Whether scrapy-redis can handle this situation - request urls have priority? Such as I need a request URL_1 to get the basic info and another request URL_2 to get the detail (URL_1 has to before URL_2)?

Thanks a lot.

sunnypandas avatar Apr 09 '18 14:04 sunnypandas

It doesn't. You would need to implement a priority queue in redis as backend.

rmax avatar Apr 09 '18 14:04 rmax

Thank you @rmax firstly.

This is really a problem in distributed system.

sunnypandas avatar Apr 10 '18 00:04 sunnypandas