JohnMTrimbleIII

Obey Robots.txt

9

Is scrapy-splash not compatible with obeying robots.txt? Everytime I make a query it attempts to download the robots.txt from the docker instance of scrapy-splash. The below is my settings file....

Multiple Changes

2

Is there a way to build a queue of drafts for a single record? I know the behavior is that there is only one draft per record, do new drafts...

I'm trying to follow the into documentation. I changed the training url to be an https one and get the following. > Traceback (most recent call last): > File "",...

How to make scrapy-splash work with frontera?

2

I have added the following lines to my spider. ```` def start_requests(self): for url in self.start_urls: yield SplashRequest(url, self.parse, args={ # optional; parameters passed to Splash HTTP API 'wait': 0.5,...