crawlee
crawlee copied to clipboard
refactor: confusing timeout handling
Problem:
Confusing timeout logic that causes a user sees a 60s request handler timeout, but the errors will mention 130s (or 100s for HTTP crawlers where navigation timeout is only 30s).
What this PR changes
- Refactors _handleNavigation in http-crawler.ts and browser-crawler.ts so the navigation timeout covers the entire navigation flow, including navigation hooks.
- Removes the buffer timeout in both crawlers.
- Improves BasicCrawler messaging: instead of “requestHandler timed out after 100s/130s,” the error now states that a safety timeout was reached.
- This safety timeout is disabled by default with the new logic, but can be re-enabled by setting it to any value > 0.
Impact / Compatibility
- Workloads that implicitly relied on the hidden buffer may now time out sooner. We may want to increase the default navigation timeout to compensate.
- No behavioral surprises beyond the tighter, more honest timing.
Closes #2951
Contributors: @ezequiel38 @anghel9