scrapy-proxies icon indicating copy to clipboard operation
scrapy-proxies copied to clipboard

Feature: Proxy mode: change proxy every N-th request

Open exactstat opened this issue 6 years ago • 1 comments

If it's possible, please, add a possibility to change proxy every N-th request. Add a variable (for setting N) and a new value for "Proxy mode" for this.

exactstat avatar Jun 23 '18 14:06 exactstat

Some services like Luminati provide a single endpoint for all requests.

The right approach is to re establish the connection (after n connections, or in case of failure) to the same endpoint.

Currently, if the endpoint gets blocked, the crawl will be terminated because there's no more proxies available.

If I try to copy paste the proxy multiple times in the file, is still being aggregated in the code.

Lusitaniae avatar May 17 '20 06:05 Lusitaniae