purple-hats icon indicating copy to clipboard operation
purple-hats copied to clipboard

Published 20 hours ago • verified publisher icon GovTechSG

BasicCrawler limit

Open mgifford opened this issue 3 years ago • 1 comments

Is there a way to get around this?

INFO: BasicCrawler: Crawler reached the maxRequestsPerCrawl limit of 100 requests and will shut down soon. Requests that are in progress will be allowed to finish.
INFO: BasicCrawler: Earlier, the crawler reached the maxRequestsPerCrawl limit of 100 requests and all requests that were in progress at that time have now finished. In total, the crawler processed 104 requests and will shut down.

mgifford avatar Sep 04 '20 13:09 mgifford