frontera icon indicating copy to clipboard operation
frontera copied to clipboard
verified publisher icon scrapinghub

frontera is being polite to my splash server. How to disable that?

Open casertap opened this issue 9 years ago • 2 comments

It seems that frontera is being polite to my splash server and I obviously does not want that. Frontera is looking for the robot.txt `2016-09-23 13:08:10 [scrapy] DEBUG: Crawled (404) <GET http://splash:8050/robots.txt> (referer: None)`` Frontera is also applying the frontier rules to my internal splash server.

How can I disable that?

casertap avatar Sep 24 '16 00:09 casertap

Please send your Scrapy spider settings. It's hard to guess how you generate Splash requests. I would expect Scrapy is generating robots.txt and it's managed with http://doc.scrapy.org/en/latest/topics/settings.html#robotstxt-obey

sibiryakov avatar Sep 26 '16 20:09 sibiryakov

what do you call frontier rules?

sibiryakov avatar Sep 26 '16 20:09 sibiryakov