crawlee icon indicating copy to clipboard operation
crawlee copied to clipboard

refactor: confusing timeout handling

Open anghel9 opened this issue 4 months ago • 0 comments

Problem:

Confusing timeout logic that causes a user sees a 60s request handler timeout, but the errors will mention 130s (or 100s for HTTP crawlers where navigation timeout is only 30s).

What this PR changes

  • Refactors _handleNavigation in http-crawler.ts and browser-crawler.ts so the navigation timeout covers the entire navigation flow, including navigation hooks.
  • Removes the buffer timeout in both crawlers.
  • Improves BasicCrawler messaging: instead of “requestHandler timed out after 100s/130s,” the error now states that a safety timeout was reached.
  • This safety timeout is disabled by default with the new logic, but can be re-enabled by setting it to any value > 0.

Impact / Compatibility

  • Workloads that implicitly relied on the hidden buffer may now time out sooner. We may want to increase the default navigation timeout to compensate.
  • No behavioral surprises beyond the tighter, more honest timing.

Closes #2951

Contributors: @ezequiel38 @anghel9

anghel9 avatar Aug 12 '25 23:08 anghel9