crawlee

crawlee copied to clipboard

Enable proper forefront for request queue when used together with request list

1

**Describe the bug** This is somewhere between a bug and a feature request. When you use RequestList and RequestQueue together, all requests are first taken from the list before any...

metalwarrior665

**Describe the feature** Allow `errorHandler` to work with the full crawling context. For example, make sure that the `page` is not closed before `errorHandler` finishes. We should also prevent errors...

mnmkng

Apify cli command to put failed requests back to queue

12

**Describe the feature** There was a time in beta, handled and pending request in queue were in JSON format. If we wanted to retry some failed requests, we can simply...

hungnv-sr

fix(pseudourl): support escaping and check for unbalanced pairs of brackets

5

This _could_ be a breaking change: pseudourl `https://example.com/]` was valid until now. If that is unacceptable, I can revert that part of the changes (and we could maybe only keep...

mvolfik

* feat rust high performance efficient parallel crawling start This pr starts the integration with a native high performance gRPC crawler that is the fastest and most efficient OSS indexer...

j-mendez

[WIP] Request queue v2 - batch operations, locking, requests list

3

Endpoints for the request queue v2 project are ready with apify-client code. This issue should cover what needs to be done on the SDK site to get the request queue...

drobnikj

CheerioCrawler cookies not set by 302 redirect response. Expected?

9

**Describe the bug** When a CheerioCrawler request results in a redirect, the set-cookie header from the 302 response is not put into the cookie header of the subsequent request to...

justingrant

Add option to deactivate auto-retire when proxy is blocked

2

**Describe the feature** In Playwright/Puppeteer crawler, when response is for example 403, crawler automatically throw `Error: Request blocked - received 403 status code.`. Please add option to disable this functionality...

rubydev

Crawlers should have an option to respect robots.txt

5

Now you have to write your own function to parse and respect target website's robots.txt file. Common function in an SDK (utils.js probably) for that would be great.

jakubbalada

**Describe the feature** Use `handlePageFunction` as the reference when the hooks happen. `postNavigationHooks` sound like it will be executed after `handlePageFunction` and `preNavigationHooks` before `handlePageFunction` but not just after opening...

pocesar