💡 Feature request: ROBOTS_DISABLE_INDEXING should use noindex

[kotx]

Is your feature request related to a problem? Please describe.

Certain search engines don't respect robots.txt and only follow noindex.

Describe the feature you would like to be implemented

Inject

<meta name="robots" content="noindex, nofollow">

into the head of every page served.

Describe alternatives you've considered

Manually implementing user-agent/IP blacklisting on a reverse proxy.

Additional context / screenshot

Example: https://safe.search.brave.com/help/brave-search-crawler