scrapyrt
scrapyrt copied to clipboard
scrapinghub
HTTP API for Scrapy spiders
Allow multiple urls crawling within a single API request. I need to be able to parse multiple url with a single request, so I've decided to separate them with a...
Hi, scrapyrt works very well for me, but I would like to use it with the autothrottle extension. Why is it disabled by default? Is there an easy way to...
it will default to logging.DEBUG but users can set this to something higher in case they dont want all the noise. fixes #30 for future we can also add command...
Collects not only spider exceptions, but also other kinds of errors that appear in logs. E.g.: ``` { "status": "ok", "errors": [ "Ignoring response : HTTP status code is not...
We should add some command line argument to allow to customize logging, this would be useful for running scrapyrt in production but also good for apps that inherit from scrapyrt...
I'm experiencing difficulties in accessing a ScrapyRT service running on specific ports within a Kubernetes pod. My setup includes a Kubernetes cluster with a pod running a Scrapy application, which...
Also reflect Python 3.12 support; add some env vars default fallbacks for common CLI args; and fix a `os.path.join` to be properly cross-platform As for the `package` CLI arg, you...
Are there any specific features or bug fixes that you (users, developers, anyone) consider important, top priority? Please share information here in thread. If someone would like to volunteer to...
As of January 2023, is this still maintained or is it safe to use in its current state?
Hi! thanks for your work on Scrapyrt! I've discovered that spiders served by Scrapyrt don't save the output in the Spider's / custom_settings / FEEDS. Is it possible to change...
