scrapy-rotating-proxies icon indicating copy to clipboard operation
scrapy-rotating-proxies copied to clipboard

how to ban the proxy with timeout?

Open leolle opened this issue 3 years ago • 1 comments

I have the global DOWNLOAD_TIMEOUT in setting for 3 seconds, how to ban the proxy if timeout happens? Thank you.

PS, I can see the proxy with connection timeout is active all the time.

leolle avatar Nov 11 '21 14:11 leolle

@leolle add this method to your spider

from twisted.internet.error import TimeoutError

def exception_is_ban(self, request, exception):
        return isinstance(exception, TimeoutError)

cpouldev avatar Mar 28 '22 20:03 cpouldev