spidermon icon indicating copy to clipboard operation
spidermon copied to clipboard

Add `JobsComparisonMonitor`

Open tcurvelo opened this issue 3 years ago • 2 comments

closes #301

It adds the JobComparisonMonitor, for checking for drops in scraped item count compared to previous jobs, when running jobs in Scrapy Cloud.

  • allow filtering by jobs states (eg.: finished, running, closespider_timeout, etc)
  • allow filtering by arbitrary tags
  • also updates the module that holds a singleton to the Scrapy Cloud API client.

TODO:

  • [x] use scrapinghubclient instead
  • [x] inherit from BaseStatMonitor
  • [x] filter by tags

tcurvelo avatar Nov 29 '21 02:11 tcurvelo

Codecov Report

Merging #323 (d3f5a7f) into master (06d70af) will increase coverage by 0.58%. The diff coverage is 88.23%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #323      +/-   ##
==========================================
+ Coverage   74.20%   74.79%   +0.58%     
==========================================
  Files          68       68              
  Lines        3032     3067      +35     
  Branches      462      469       +7     
==========================================
+ Hits         2250     2294      +44     
+ Misses        719      705      -14     
- Partials       63       68       +5     
Impacted Files Coverage Δ
spidermon/utils/zyte.py 84.00% <84.00%> (ø)
spidermon/contrib/scrapy/monitors.py 97.57% <94.11%> (-0.69%) :arrow_down:
spidermon/contrib/scrapy/extensions.py 85.26% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 06d70af...d3f5a7f. Read the comment docs.

codecov[bot] avatar Dec 01 '21 02:12 codecov[bot]

@rennerocha hi! Bothering you again 🙈 This PR is currently stalled, but the related ticket is assigned for the next milestone so I imagine we'd like to move forward with it. Do you think it would be ok to assign someone from Zyte to work on it? If you'd like to continue with it instead no worries, we can work on something else, let us know what you'd like to do!

curita avatar Aug 03 '22 01:08 curita