scrapy icon indicating copy to clipboard operation
scrapy copied to clipboard

add sticky meta spider middleware

Open Granitosaurus opened this issue 6 years ago • 3 comments

This change adds sticky meta spider middleware for enabling sticky meta behaviour as described in issue https://github.com/scrapy/scrapy/issues/3645

Fixes #3645

StickyMeta middleware transfers specified meta keys to further
requests in spider request chain.

Sticky meta keys can be configured either by:
* Setting STICKY_META_KEYS eg:
    STICKY_META_KEYS = ['key1', 'key2']
* Spider attribute sticky_meta eg:
    spider.sticky_meta = ['key1','key2']
* Meta attribute "sticky" eg:
    Request(meta={'sticky': ['foo'], 'foo': 'bar'})
resolve priority is meta > spider attribute > setting

Granitosaurus avatar May 10 '19 14:05 Granitosaurus

Codecov Report

Merging #3770 into master will decrease coverage by 0.53%. The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master    #3770      +/-   ##
==========================================
- Coverage   85.42%   84.89%   -0.54%     
==========================================
  Files         169      170       +1     
  Lines        9635     9656      +21     
  Branches     1433     1439       +6     
==========================================
- Hits         8231     8197      -34     
- Misses       1156     1204      +48     
- Partials      248      255       +7
Impacted Files Coverage Δ
scrapy/spidermiddlewares/stickymeta.py 85.71% <85.71%> (ø)
scrapy/core/downloader/handlers/s3.py 62.9% <0%> (-32.26%) :arrow_down:
scrapy/utils/boto.py 46.66% <0%> (-26.67%) :arrow_down:
scrapy/core/downloader/tls.py 77.5% <0%> (-12.5%) :arrow_down:
scrapy/extensions/feedexport.py 80.18% <0%> (-4.72%) :arrow_down:
scrapy/core/downloader/handlers/http11.py 89.92% <0%> (-2.62%) :arrow_down:
scrapy/core/scraper.py 86.48% <0%> (-2.03%) :arrow_down:
scrapy/pipelines/files.py 67.15% <0%> (-1.1%) :arrow_down:

codecov[bot] avatar May 10 '19 14:05 codecov[bot]

Here is an implementation https://github.com/heylouiz/scrapy-sticky-meta-params

ejulio avatar Oct 28 '19 18:10 ejulio

New PR for the topic #5852 based on @heylouiz repo

felipeboffnunes avatar Mar 16 '23 14:03 felipeboffnunes