scrapyrt
scrapyrt copied to clipboard
scrapinghub
HTTP API for Scrapy spiders
Basically I want to prevent unauthorized clients from accessing the scrapyrt API. I would want to secure a scrapyrt API, is there anything built in handling an authorization mechanism ?...
Hello, I'm using scrapyrt to provide an HTTP interface to a big Scrapy project. I'm running a single scrapyrt instance in a Docker container, some spiders require ~60-120 seconds to...
(Sorry can't find how to label this) I hope this is the right place where to ask this. I created a spider that can scrape a page in an e-commerce...
@pawelmhm requesting ability to scrape multiple spiders asynchronously at once instead of overwhelming the server with request Here is what i mean: ```python { "request": { "url":["https://www.site1.com","https://www.site2.com","https://www.site3.com"] , "callback": "parse_product",...
There is not much about deploying scrapyrt, there is only one short section in wiki. There should be more info in readthedocs.
I have been using scrapyrt on a live project, it works perfectly fine and fulfils all it's purposes. But sometimes it just stops responding, I run scrapyrt with systemd service...
`max_items` like the `max_requests` argument would be really helpful.
I have been exploring many options on how to keep scrapyrt open and active even after reboot, but I am unsure what is best. I was thinking of using [immortal.run](https://immortal.run/)....
Actually i read scrapyrt documentation and its mention that i need to modify ScrapyRT and need to use websockets or HTTP push notifications.But i don't understand how to do this?...I...
