Need to add `/robots.txt` to gateways

[jbenet]

would be nice if gateways handled:

> curl 127.0.0.1:8080/robots.txt
User-agent: *
Allow: /

notes:

• allow or disallow API?
• this should be optional (config var?)
• this should redirect robots to our main gateway https://ipfs.io to have a canonical location for all URLs found that will be properly rerouted with browser implementations/extensions.

[jbenet]

from https://github.com/ipfs/go-ipfs/pull/1747/files#r40535116:

• [ ] consider how this plays with header from ipfs/kubo#1747
• [ ] perhaps make an option like Gateway.Robots.AllowIndexing, default to false.
• [ ] option drives this header, and /robots.txt.
• [ ] sharness tests for the config option, expected header, and /robots.txt