Andrey Rakhmatullin

Refactor/clean up `handle_httpstatus_list`/`HTTPERROR_ALLOWED_CODES` etc.

1

There are several related things used by `HttpErrorMiddleware` and `RedirectMiddleware` (`handle_httpstatus_list` spider attr and meta key, `handle_httpstatus_all` meta key, `HTTPERROR_ALLOWED_CODES` and `HTTPERROR_ALLOW_ALL` settings), it would be nice to unify these,...

[PoC] Reactorless tests.

1

This adds a new mode of running tests, one that disables pytest-twisted and enforces that there is no reactor. It doesn't use pytest-asyncio for now, instead skipping all async test...

Support `pyproject.toml` as a preferred alternative to `scrapy.cfg`

1

Suggested by @Gallaecio

We deprecated almost all spider attrs used instead of settings, but `download_delay` is used to set the delay calculated by `AutoThrottle` so it can't be deprecated right now, but it...

Release notes for 2.14.0

1

Preliminary release notes for master, to get an idea etc.

Discourage using `CrawlSpider`

4

I've seen a lot of user questions about `CrawlSpider`, most of them caused by its limitations. I'm biased but when asked I awlays reply "please consider not using it". In...

Update the docs about global settings

3

We should clarify https://docs.scrapy.org/en/master/topics/settings.html#special-settings based on the analysis in https://github.com/scrapy/scrapy/pull/6568#discussion_r1925145591 and ideally also make sure all affected settings themselves also say, or link, their specifics in their docs.

Refactor autouse pytest.skip() fixtures in conftest.py

1

I think fixtures that check pytest marks and conditionally call `pytest.skip()` can be combined into one or two (one for the reactor, one for optional test deps?) and be some...

Can be done now for `process_item()`, but `{open,close}_spider()` use a `DeferredList` and so it's better to refactor that first

Emit deprecation warnings for Deferreds passed to async APIs that support coroutines

1

We have (and should make more) APIs that take both Deferreds and coroutines and wrap them in `deferred_from_coro()`, e.g. downloader middleware and pipeline methods. We should emit deprecation warnings if...