crawlee

crawlee copied to clipboard

feat: add `keepAlive` to crawler options

5

Allows to keep the crawler running even when the queue is empty. Use `crawler.teardown()` to stop it. Closes #1436

B4nan

Benchmarks (100 `crawlee.dev` URLs): - `HttpCrawler`: 1.5s - `CheerioCrawler`: 4s - `LinkeDOMCrawler`: 8s Why not JSDOM: https://github.com/jsdom/jsdom/issues/2005 IOW it's very slow Why not happy-dom: no idea if it's possible JS...

szmarczak

The PR adds the error to the crawling context for the error handlers. Since you can throw *anything*, the correct type for errors in the error handlers is unknown instead...

vladfrangu

**Describe the feature** Sometimes we want to keep the crawler alive even if the request queue is empty. Current workaround is to override the `isFinishedFunction` of `AutoscaledPool`

B4nan

[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [ws](https://togithub.com/websockets/ws) | [`^7.5.9` -> `^8.0.0`](https://renovatebot.com/diffs/npm/ws/7.5.9/8.8.1) | [](https://docs.renovatebot.com/merge-confidence/)...

renovate[bot]

chore(deps): update dependency puppeteer to v16

1

[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [puppeteer](https://togithub.com/puppeteer/puppeteer) | [`>= 9.x ` `16.1.0`](https://renovatebot.com/diffs/npm/puppeteer/14.4.1/16.1.0) |...

renovate[bot]

Allow checking `INPUT.json` file directly in the project root

5

``./apify_storage/key_value_stores`` is a generated dir I think the path of ``INPUT.json`` should be ``./INPUT.json``

hugetiny

**Describe the bug** When first async storage method is invoked, the default storages are automatically purged. If there are more racing promises (e.g. via `Promise.all`) doing this, there is a...

B4nan

**Describe the feature** Add a list of error messages and count of how much they occurred in the `Statistics` object. Let's start with storing 100 messages, trimming to the first...

B4nan

Improve HTTP status code handling

4

There are several issues now which are related to the way we handle HTTP status codes in crawlers. - `CheerioCrawler` throws an exception when it encounters a 500+ status code...

mnmkng