frontera
frontera copied to clipboard
Get rid of OverusedBuffer
Buffering requests to busy hosts should be responsibility of fetcher component. We need to figure out how to change interfaces, and how to support necessary buffering logic in our default fetcher (Scrapy).
That would require talking directly with Scrapy Downloader, which isn't possible using Scrapy Scheduler interface.