[Feature] add robots.txt to headscale

[ykuksenko]

Use case

Tailscale derp servers serve a robots.txt to avoid being hit by crawlers. Headscale should probably do this too.

see derp servers list or this derp server /robots.txt to verify my claim.

Description

$HEADSCALE_URL/robots.txt should be populated with:

User-agent: *
Disallow: /

Contribution

• [ ] I can write the design doc for this feature
• [ ] I can contribute this feature

How can it be implemented?

No response

edit: This may be needed for the server itself vs just derp - not sure where the line is for that with headscale. edit2: I think I can submit a PR for this from a quick glance at the code.

[kradalby]

Good idea, PR would be appreciated!