Add Crawl-delay to robots.txt by default

[touzoku]

We have recently faced a DDoS attack on our ghost instance using a search bot amplification mechanism. This setting is respected by some search engines such as Bing and Yandex and mitigates such attacks. This should be a default setting, we believe.

[codecov[bot]]

Codecov Report

Merging #14989 (4024ee6) into main (15294f4) will decrease coverage by 0.00%. The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #14989      +/-   ##
==========================================
- Coverage   61.54%   61.53%   -0.01%     
==========================================
  Files         573      573              
  Lines       46430    46430              
  Branches     4209     4209              
==========================================
- Hits        28575    28571       -4     
- Misses      17809    17813       +4     
  Partials       46       46              

Impacted Files	Coverage Δ
core/server/models/base/plugins/events.js	69.48% <0.00%> (-1.48%)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 15294f4...4024ee6. Read the comment docs.

[ErisDS]

Hey @touzoku, I'm sorry for taking so long to respond & also very sorry to hear you've had DDoS issues. I can imagine that was pretty stressful.

It's possible to override Ghost's default robots.txt file by adding your own file to the root of your theme, which should at least allow you to mitigate the issue.

I'm all for improving the defaults in Ghost, but before merging this PR I'd love to see some reference material for why this specific setting should be Ghost's default. Did you find any good information about how various bots respect the setting, and what the right value is to balance timely updates and mitigating attacks?

[github-actions[bot]]

Hey @touzoku 👋

Our team needed some more info to get to the bottom of this, however we've not heard back from you. We're going to close this for now, but let us know if you manage to dig up some more info and we'll reopen.