Michael McMahon
Michael McMahon
I see. I'm glad I'm not just messing up the config. If I find someone willing to develop this feature, would the inclusion of this feature be likely added if...
Cool. That helps! I'll see what I can drum up.
I have asked around, but no one seemed interested enough to jump on it.
@bigalownz Can you enable your issue tracker?
Those are good questions. The technology exists today for free desktop OSes, but it is still in the developer skill-set range and not a user-friendly range. The above script could...
My heart rate increased reading these issues. Scrapy doesn't bother with Crawl-Delay. I am seeing users in the wild skipping robots.txt entirely. I have not seen any contact information from...
[You have essentially created a death star that people are actively using to take down sites.](https://imgflip.com/i/9wcu7h)  If your service uses better defaults, apply those defaults to the project. Add...
Tested on 0.96.3 and issue persists.
I can help write something up from this.
Without JS that other page is not blank, but it is still not very helpful. It says: ``` Please verify to continue I'm not a robot. Verification Instructions 1. Press...