scrapy-rotating-proxies icon indicating copy to clipboard operation
scrapy-rotating-proxies copied to clipboard

Published 20 hours ago • verified publisher icon TeamHG-Memex

Retry Request

Open kadimon opened this issue 5 years ago • 1 comments

Hello! Can you show me an example of a policy that will re-try to visit the page?

kadimon avatar Jun 21 '19 16:06 kadimon

I know it's a late reply, but for future readers: the only thing a ban policy does is answering the two following questions:

  • "Are we banned, judging by the response?" (response_is_ban)
  • "Are we banned, judging be the exception?" (exception_is_ban).

It is then BanDetectionMiddleware's job to mark the response as banned and then it is RotatingProxyMiddleware's job to retry a request in case of ban.

StasDeep avatar Jun 07 '20 08:06 StasDeep