spidermon

spidermon copied to clipboard

docs: Documenting built-in mixins

2

Continue work started on https://github.com/scrapinghub/spidermon/pull/133

rennerocha

feature: Added Cerberus as a new option for item validation

1

:hatching_chick: This brings an end to a great and fulfilling period of contributing to Spidermon and the Scrapy Project as part of Google Summer of Code 2019. # **Google Summer...

vipulgupta2048

Create tests for PythonExpressionsMonitor

4

This feature is not tested: https://github.com/scrapinghub/spidermon/blob/master/spidermon/python/factory.py#L12

rennerocha

This bit needs to be documented in the code https://github.com/scrapinghub/spidermon/blob/54b8e873aee2c60c039dcccab2b0dd8fcd502aa0/spidermon/core/monitors.py#L10

vipulgupta2048

no proper message or monitors for extra fields when using jsonschema

1

The stats for unexpected fields is almost unreadable and can't be used for writing a monitor to raise error if a crawler is crawling any unexpected field. `'spidermon/validation/fields/errors/unexpected_fields_upopularity_ranking_uratings_utotal_competition_ucertificate_of_excellence_unum_of_photos_uclaimed_uprice_range_were_unexpected': 1, 'spidermon/validation/fields/errors/unexpected_fields_upopularity_ranking_uratings_utotal_competition_ucertificate_of_excellence_unum_of_photos_uclaimed_uprice_range_were_unexpected/':...

HassanQamar07

I was trying to compare spider result on scrapyinghub and failed

2

### I got two problems: 1 . Get `SpidermonHistory Validation/Expected number of items extracted... ERROR.` ( already config STATS_CLASS on hub) ``` python spider = self.data["spider"] print(spider.stats_history) ``` 2. I...

Kiollpt

I had difficulty figuring out how to use jsonschema

2

Following https://spidermon.readthedocs.io/en/latest/item-validation.html#with-json-schema I set up ITEM_PIPELINES then jumped down to the jsonschema section, but there are no settings listed there for how to use jsonschema. After that I scanned the...

andrewbaxter

Option to stop spider on errors

15

I think it would be nice to have validation errors treated like other spider errors - increment the error counter, optionally stop the spider with an error condition if over...

andrewbaxter

Currently, to send _SES_ emails, we need to set `SPIDERMON_AWS_ACCESS_KEY_ID` and the other configs for that. Though, it may be the case of a single user configured in scrapy for...

ejulio

Part of my GSoC project. This implements the CLI and the commands `version` that shows Spidermon's installed version, and `setup`, that enables Spidermon if it's not enabled, and the monitors...

leovictorsr