[Bug]: Search engines index non-local communities, leading to undesirable results #3098

[binwiederhier]

Summary

Thank you for your fantastic work on Lemmy. I love it!

Issue Summary

Due to the nature of the default robots.txt and the meta tags in Lemmy, search engines will index even non-local communities. This leads to results that are undesirable, such as unrelated/undesirable content being associated with your instance.

Example:

Suggested remediation/feature

I think it should be an opt-in feature to have non-local communities be indexed, e.g. [ ] Allow search engines to index non-local communities

Temporary workaround

I added this to my nginx config to prevent search engines from indexing the entire site:

# Disallow all search engines
location / {
  ...
  add_header X-Robots-Tag noindex;
}

location = /robots.txt {
    add_header Content-Type text/plain;
    return 200 "User-agent: *\nDisallow: /\n";
}

(I originally reported in https://github.com/LemmyNet/lemmy/issues/3098, but this seems to be the right place for it.)

Steps to Reproduce

Open Google, and type <your instance> north korea. You'll see a bunch of posts related to north korea that are not related to any of your local communities.

Technical Details

n/a

Version

0.17.4