nginx-ultimate-bad-bot-blocker icon indicating copy to clipboard operation
nginx-ultimate-bad-bot-blocker copied to clipboard

Published 20 hours ago • verified publisher icon mitchellkrogza

[CCBot/2.0 (http://commoncrawl.org/faq/)] (An excellent nonprofit project that is blocked)

Open rbjarnason opened this issue 2 years ago • 0 comments

Paste the full User-Agent String here

CCBot/2.0 (http://commoncrawl.org/faq/)

Is this for Addition / Removal?

  • [ ] Addition
  • [x] Removal
  • [ ] Keep a watch on this one

Did the User-Agent request robots.txt first?

  • [ x] Yes
  • [ ] No

Additional information

Thank you for your excellent project that has been a great help to us, but we also noticed you blocked CCBot. CommonCrawl is an excellent nonprofit project that we use a much ourselves. CCBot respects both robots.txt and Crawl-Delay.

rbjarnason avatar Sep 10 '22 15:09 rbjarnason