spidermon
spidermon copied to clipboard
scrapinghub
Scrapy Extension for monitoring spiders execution.
## Description When using the `SendSmtpEmail` class from Spidermon, even though the email is sent successfully, an error is thrown by the Twisted framework. I followed the syntax from the...
Ubuntu 22.04, Python 3.10.9, Spidermon 1.17.1 If an item defined with `attrs` library and this condition https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/pipelines.py#L140 is `True`, then it fails here https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/pipelines.py#L141 with this message: ``` Traceback (most...
Considering that Spidermon was created at Zyte (former Scrapinghub), it is understandable that we have monitors and actions that are related to services provided by it. However, considering that Spidermon...
## Background All Spidermon failed checks are reported as ERRORs at the moment. Everything belongs to the same category and can't be addressed differently. It could be nice to have...
Hi Spidermon team, Was looking over the schematics project and noticed it looks like it is no longer maintained. I note that it doesn't support Python 3.10 and doesn't appear...
## Background FieldCoverageMonitor is configured via the `SPIDERMON_FIELD_COVERAGE_RULES` setting. This dictionary defines how much coverage we expect from each field (expressed as a float number between 0 and 1, 1...
Adds limit for nested dict stats computation Added a new setting `SPIDERMON_JOBS_COMPARISON_CLOSE_REASONS` to allow `ZyteJobsComparisonMonitor` to filter by the ScrapyCloud jobs with their `close_reason` stat.
## Background The method to get the previous jobs used in ZyteJobsComparisonMonitor can be seen here: https://github.com/scrapinghub/spidermon/blob/a7e195f951a4b9a4837943dff0b2ae727b76f237/spidermon/contrib/scrapy/monitors/monitors.py#L557-L579 where `number_of_jobs`'s value is defined by the `SPIDERMON_JOBS_COMPARISON` setting. ## Issue Supposedly, we...
## Background ZyteJobsComparisonMonitor is a monitor that compares `item_scraped_count` from the current job against past jobs from ScrapyCloud. By default, it will grab `#SPIDERMON_JOBS_COMPARISON` past jobs, and these jobs can...
## Background ZyteJobsComparisonMonitor is a monitor that compares item_scraped_count from the current job against past jobs from ScrapyCloud. By default, it will grab `#SPIDERMON_JOBS_COMPARISON` past jobs, and these jobs can...
Metadata
Owner
Metadata
Scrapy Extension for monitoring spiders execution.