spidermon
spidermon copied to clipboard
Add `JobsComparisonMonitor`
closes #301
It adds the JobComparisonMonitor
, for checking for drops in scraped item count compared to previous jobs, when running jobs in Scrapy Cloud.
- allow filtering by jobs states (eg.:
finished
,running
,closespider_timeout
, etc) - allow filtering by arbitrary tags
- also updates the module that holds a singleton to the Scrapy Cloud API client.
TODO:
- [x] use scrapinghubclient instead
- [x] inherit from
BaseStatMonitor
- [x] filter by tags
Codecov Report
Merging #323 (d3f5a7f) into master (06d70af) will increase coverage by
0.58%
. The diff coverage is88.23%
.
@@ Coverage Diff @@
## master #323 +/- ##
==========================================
+ Coverage 74.20% 74.79% +0.58%
==========================================
Files 68 68
Lines 3032 3067 +35
Branches 462 469 +7
==========================================
+ Hits 2250 2294 +44
+ Misses 719 705 -14
- Partials 63 68 +5
Impacted Files | Coverage Δ | |
---|---|---|
spidermon/utils/zyte.py | 84.00% <84.00%> (ø) |
|
spidermon/contrib/scrapy/monitors.py | 97.57% <94.11%> (-0.69%) |
:arrow_down: |
spidermon/contrib/scrapy/extensions.py | 85.26% <100.00%> (ø) |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 06d70af...d3f5a7f. Read the comment docs.
@rennerocha hi! Bothering you again 🙈 This PR is currently stalled, but the related ticket is assigned for the next milestone so I imagine we'd like to move forward with it. Do you think it would be ok to assign someone from Zyte to work on it? If you'd like to continue with it instead no worries, we can work on something else, let us know what you'd like to do!