frontera icon indicating copy to clipboard operation
frontera copied to clipboard

Get rid of OverusedBuffer

Open sibiryakov opened this issue 9 years ago • 1 comments

Buffering requests to busy hosts should be responsibility of fetcher component. We need to figure out how to change interfaces, and how to support necessary buffering logic in our default fetcher (Scrapy).

sibiryakov avatar Jan 27 '16 17:01 sibiryakov

That would require talking directly with Scrapy Downloader, which isn't possible using Scrapy Scheduler interface.

sibiryakov avatar Feb 15 '16 12:02 sibiryakov