crawlee icon indicating copy to clipboard operation
crawlee copied to clipboard

Unify timeouts throughout our classes

Open mnmkng opened this issue 6 years ago • 0 comments

PuppeteerCrawler

  • gotoFunction has a constant timeout inside, which can be overridden by overriding the function.
  • handlePageFunction has its own timeout.

CheerioCrawler

  • prepareRequestFunction does not have a timeout.
  • handlePageFunction has its own timeout.

BasicCrawler

  • handleRequestFunction has its own timeout. When using Puppeteer or Cheerio, the timeout is set to a multiple of their handlePageFunction.
  • handleFailedRequestFunction does not have a timeout.

AutoscaledPool

  • has no timeouts.

PuppeteerPool

  • has puppeteerOperationsTimeoutSecs for puppeteer related stuff.

It's a mess.

mnmkng avatar Mar 14 '19 14:03 mnmkng