Kevin Lloyd Bernal

refactor LogStats extension to log IPM and RPM to stats on spider_close

3

Fixes #4110 **NOTE:** I noticed that the `LogStats` extension doesn't have any tests at all. I will write the tests later when this refactoring approach is approved.

Support more secure ways to declare the APIKEY

1

### BACKGROUND: As of version `1.6.0`, there are two (2) ways of adding the API KEYS: 1. via the `settings.py`: ``` CRAWLERA_APIKEY = 'apikey' ``` 2. via spider attribute: ```...

Discussion for supporting Scrapy's LinkExtractor

10

One neat feature inside Scrapy is it's [LinkExtractors](https://github.com/scrapy/scrapy/blob/64905e3397a5b837312169a0b418857ef1cf40c7/scrapy/linkextractors/lxmlhtml.py) functionality. We usually try to use this whenever we want links to be extracted inside a given page. Inside **web-poet**, we can...

This PR attempts to scrutinize the idea as noted down in https://github.com/scrapinghub/web-poet/pull/42#issuecomment-1141831361. It's built on top of this PR's branch: - https://github.com/scrapinghub/web-poet/pull/42

Introduce alternative constructors to handle nested dependencies [migrate logic from scrapy-poet]

1

### Background Given the following PO structure below: ```python import attr from web_poet.pages import Injectable from web_poet.page_inputs import ResponseData @attr.define class HTMLFromResponse(Injectable): response: ResponseData @attr.define class WebPage(Injectable): response: ResponseData @attr.define...

Proposal: Utility functions that interacts with the rules

2

## Background Following the acceptance of https://github.com/scrapinghub/web-poet/pull/27, developers could now use URL patterns to declare which Page Objects would work on specific URL patterns ([reference code](https://github.com/scrapinghub/web-poet/pull/27/files#diff-28986a071b3d856ca1ef15cb80b1390bdb551aef42f5caffa4d7d12acfeae301R57-R80)). ## Problem For large...

Stemming off from this PR: https://github.com/scrapinghub/web-poet/pull/22 The main motivation for this is for developers to easily create `HttpRequestBody` and `HttpRequest` instances from both **bytes** and **unicode strings**.

The changes from this PR was taken directly from https://github.com/scrapinghub/web-poet/pull/23 in order to have a separate discussion regarding how to better validate the Meta values (and keys). The original discussion...

add core exceptions

1

Built on top of https://github.com/scrapinghub/web-poet/pull/22. --- - [ ] Docs - [ ] Changelog - [ ] Move existing `RequestBackendError` to this new module

POP - fundamental docs

1

Built on top of the branch from https://github.com/scrapinghub/web-poet/pull/27 which aims to define the minimum specifications/requirements to be a POP.