(Feature) Configurable allow/deny search indexing via robots.txt

[l3pr-org]

Summary

It may be advantageous to, by default, disable search indexing of the OneTimeSecrets web application. Currently, Google (and others) are indexing the site, potentially making it very easy for an attacker to collect a list of instances. I propose that search indexing be opt-in-- the default behavior of this configuration option should be to completely disallow indexing in order to create more privacy-respecting defaults.

Examples

Searches that could be used to find instances:

https://www.google.com/search?q=%22Powered+by+OTS+v1.17.3-2-g046bc25%22

and

https://www.google.com/search?q=%22Create+the+secret%21%22+%2B+%22Powered+by+OTS%22

I have redacted these screenshots to retain administrator privacy if they choose to have their result(s) removed from popular search engines, however, if you wish to check-- just click on one of the provided Google queries.

Solution

A simple robots.txt to completely disallow indexing would look like:

User-agent:*
Disallow: /

After some testing, simply adding robots.txt with the above content to /frontend in this repo will work. See below screenshots.

Like I mentioned earlier, it would be even better if this feature was opt-in. I notice that configuration options are handled in /pkg/customization/customize.go, and I assume something like this would work.

...
		DisableQRSupport     bool   `json:"disableQRSupport,omitempty" yaml:"disableQRSupport"`
		DisableSearchIndex   bool   `json:"disable-search-index" yaml:"disableSearchIndex" default:"true"`
		DisableThemeSwitcher bool   `json:"disableThemeSwitcher,omitempty" yaml:"disableThemeSwitcher"`
...

Opening a PR and moving the rest of this there now that I've fleshed out the idea.

[sorcix]

This doesn't really change anything security wise. Specialized search indices like https://search.censys.io (and bad actors ignoring robots.txt) will find it anyway.

But it doesn't hurt to set a robots.txt either, I don't really see a reason to have search engines index this. There's nothing useful to find anyway.