scrapy-rotating-proxies
scrapy-rotating-proxies copied to clipboard
how to ban the proxy with timeout?
I have the global DOWNLOAD_TIMEOUT in setting for 3 seconds, how to ban the proxy if timeout happens? Thank you.
PS, I can see the proxy with connection timeout is active all the time.
@leolle add this method to your spider
from twisted.internet.error import TimeoutError
def exception_is_ban(self, request, exception):
return isinstance(exception, TimeoutError)