scrapy
scrapy copied to clipboard
add sticky meta spider middleware
This change adds sticky meta spider middleware for enabling sticky meta behaviour as described in issue https://github.com/scrapy/scrapy/issues/3645
Fixes #3645
StickyMeta middleware transfers specified meta keys to further
requests in spider request chain.
Sticky meta keys can be configured either by:
* Setting STICKY_META_KEYS eg:
STICKY_META_KEYS = ['key1', 'key2']
* Spider attribute sticky_meta eg:
spider.sticky_meta = ['key1','key2']
* Meta attribute "sticky" eg:
Request(meta={'sticky': ['foo'], 'foo': 'bar'})
resolve priority is meta > spider attribute > setting
Codecov Report
Merging #3770 into master will decrease coverage by
0.53%. The diff coverage is85.71%.
@@ Coverage Diff @@
## master #3770 +/- ##
==========================================
- Coverage 85.42% 84.89% -0.54%
==========================================
Files 169 170 +1
Lines 9635 9656 +21
Branches 1433 1439 +6
==========================================
- Hits 8231 8197 -34
- Misses 1156 1204 +48
- Partials 248 255 +7
| Impacted Files | Coverage Δ | |
|---|---|---|
| scrapy/spidermiddlewares/stickymeta.py | 85.71% <85.71%> (ø) |
|
| scrapy/core/downloader/handlers/s3.py | 62.9% <0%> (-32.26%) |
:arrow_down: |
| scrapy/utils/boto.py | 46.66% <0%> (-26.67%) |
:arrow_down: |
| scrapy/core/downloader/tls.py | 77.5% <0%> (-12.5%) |
:arrow_down: |
| scrapy/extensions/feedexport.py | 80.18% <0%> (-4.72%) |
:arrow_down: |
| scrapy/core/downloader/handlers/http11.py | 89.92% <0%> (-2.62%) |
:arrow_down: |
| scrapy/core/scraper.py | 86.48% <0%> (-2.03%) |
:arrow_down: |
| scrapy/pipelines/files.py | 67.15% <0%> (-1.1%) |
:arrow_down: |
Here is an implementation https://github.com/heylouiz/scrapy-sticky-meta-params
New PR for the topic #5852 based on @heylouiz repo