Adding arbitrary rules

[marktran]

Is it possible to add arbitrary rules? e.g. Crawl-delay:

User-agent: msnbot
Crawl-delay: 1

http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

[PikachuEXE]

user_agent :msnbot do
  add_rule 'Crawl-delay', 1
end

According to https://github.com/lassebunk/human_power/blob/master/lib/human_power/generator.rb#L27

Edit: this issue can be changed to request more doc on adding arbitrary rules

[lassebunk]

That's a good idea. I have an idea for how this could be implemented. Thanks :+1:

[PikachuEXE]

Isn't this "implemented"?

[lassebunk]

Not yet. Still thinking :blush:

[PikachuEXE]

But https://github.com/lassebunk/human_power/issues/4#issuecomment-34956234 works already?

[lassebunk]

Yes, it will work. But I'd like for you to be able to write "crawl_delay 1" (shorthand method)

[lassebunk]

Do you know of any other rules that should be added?

[PikachuEXE]

I have no idea :p @marktran you have any?

[marktran]

That's actually the only one I know of

[lassebunk]

Ok, thanks. I'll keep this in mind. Keeping this open.