phinde
phinde copied to clipboard
re-crawl on errors
When a crawl HTTP request times out, the crawler should try it again after some time automatically.
When a crawl HTTP request times out, the crawler should try it again after some time automatically.