web-poet

web-poet copied to clipboard

"Apply rules" instead of overrides

6

hey! I’m exploring how to make the following work: ```py from zyte_common_items import Product from web_poet import ItemPage, handle_urls @handle_urls("example.com") class ExamplePage(ItemPage[Product]): # ... ``` In other words, how to...

kmike

As explained in https://github.com/scrapy/w3lib/issues/189 and https://github.com/scrapy/scrapy/issues/5601, BOM should take a precedence over Content-Type headers when detecting an encoding. Currently web-poet.HttpResponse prefers Content-Type header: ```py import codecs import web_poet body =...

kmike

@field(extra=True)

1

This feature allows to declare a field as `@field(extra=True)`. Such fields are ignored if the item type doesn't have them. It allows to define fields in the base class, which...

kmike

Guidelines for using item_cls_fields: True or False by default?

2

@gatufo has some concerns about the current guidelines: https://github.com/scrapinghub/web-poet/pull/53#issuecomment-1203902479 > For me when combining both Item and Page Objects, Item defines the fields that what you want to extract, and...

kmike

Split documentation by target audience

2

At the moment we kind of mix documentation for page object writers and for framework writers. I think it would be best to split the two at the root of...

Gallaecio

Discussion for supporting Scrapy's LinkExtractor

10

One neat feature inside Scrapy is it's [LinkExtractors](https://github.com/scrapy/scrapy/blob/64905e3397a5b837312169a0b418857ef1cf40c7/scrapy/linkextractors/lxmlhtml.py) functionality. We usually try to use this whenever we want links to be extracted inside a given page. Inside **web-poet**, we can...

BurnzZ

This PR attempts to scrutinize the idea as noted down in https://github.com/scrapinghub/web-poet/pull/42#issuecomment-1141831361. It's built on top of this PR's branch: - https://github.com/scrapinghub/web-poet/pull/42

BurnzZ

w3lib-based url classes

1

An experiment: how a w3lib-based URL class could look like. Again, a minimal version.

kmike

web_poet.page_object decorator

1

This is an experiment, to see how "functional" Page Objects could look like, if it's possible to implement them, and to get feedback about the idea.

kmike

Introduce alternative constructors to handle nested dependencies [migrate logic from scrapy-poet]

1

### Background Given the following PO structure below: ```python import attr from web_poet.pages import Injectable from web_poet.page_inputs import ResponseData @attr.define class HTMLFromResponse(Injectable): response: ResponseData @attr.define class WebPage(Injectable): response: ResponseData @attr.define...

BurnzZ