spidermon
spidermon copied to clipboard
scrapinghub
Scrapy Extension for monitoring spiders execution.
The `is_url` and `is_email` in [`spidermon/contrib/validation/jsonschema/formats.py`](https://github.com/scrapinghub/spidermon/blob/ba05ad7452ad9c4851737b8c501974209a0bb945/spidermon/contrib/validation/jsonschema/formats.py) returns `True` is the input is anything but a string. Thus, `dict()`, `42`, `[42, {}, ('a', 'tuple')]` would all be valid emails or URLs...
I also added two custom periodic monitors and imported monitor ``` from spidermon.contrib.scrapy.monitors import ( ErrorCountMonitor, FinishReasonMonitor, ItemValidationMonitor, UnwantedHTTPCodesMonitor, ) ``` I didn't find the reason that got this error....
Validate maximum amount of `downloader/response_status_count/XXX` for failed responses (4xx or 5xx responses) in general. Maybe have two configurations, one for client errors (4xx) and other for server errors (5xx). If...
Make more clear in the docs https://spidermon.readthedocs.io/en/latest/actions/sentry-action.html that to enable Sentry in your monitoring, you need to create a MonitorSuite that uses SentryMessage action, or else Sentry will not receive...
Currently the notifications we receive in Slack are quite simple, without much information. For example: ``` *somesite spider finished with errors!* / view job in Scrapy Cloud _(errors=7)_ • _Job...
Hi, I'm creating a report using spidermon's CreateFileReport： ``` class SpiderCloseMonitorSuite(MonitorSuite): monitors = [ItemCountMonitor, ItemValidationMonitor, PeriodicJobStatsMonitor] monitors_finished_actions = [CreateFileReport] ``` But it turns out that Chinese is not supported，I know...
I created a Periodic Monitor to send emails if a given condition fails. However, email action fails as it reads the stat `finish_time`, but it is not set at the...
I configured a monitor as described [here](https://spidermon.readthedocs.io/en/latest/stats-collection.html) and tried to run it in scrapinghub manually. But then I get the following ERROR: ``` [root] Job runtime exception Less Traceback (most...
Scrapy has a `setting` directive implemented for Sphinx documentation that allows linking to settings while formatting them as code in an easy manner. Looking at #212, I think Spidermon could...
Metadata
Owner
Metadata
Scrapy Extension for monitoring spiders execution.