feoblog icon indicating copy to clipboard operation
feoblog copied to clipboard

Published 20 hours ago • verified publisher icon NfNitLoop

robots.txt

Open NfNitLoop opened this issue 4 years ago • 1 comments
trafficstars

FeoBlog should include a robots.txt

Immediately after sharing a link to a live server for the first time, it got hit by several bots, and they generally requested a robots.txt.

I should read up on the modern format of robots.txt and include an appropriate one.

Can they do wildcard matching? Do I want to allow/deny specific sections? (Maybe deny /client/*?)

NfNitLoop avatar Jan 21 '21 04:01 NfNitLoop

Idea:

Deny by default, but allow /u/:id/** for users "on the homepage".

Maybe also include users that list this server as their only/first server? (But I'm wary of that. I'd rather RSS-feed-type stuff not show up in indexes.)

NfNitLoop avatar Apr 26 '22 03:04 NfNitLoop