How can I disable rate limiting for google bots?

[HolyOne]

Is there a way to disable or override rate limit by user agent?

[mk1024]

You can do this by utilizing the client ID and client whitelist:

1. Use the config to set something like "ClientIdHeader": "x-is-whitelisted" and "ClientWhitelist": [ "yes" ]
2. Add your user agent check before rate limiting is applied.
3. Set the request header of every request to "yes" or "no".

You should set (and overwrite) this header for every request, so that a spoofed header value is always replaced by the result of your user agent check. At the same time, you should also keep in mind that the user agent itself can be spoofed, too :-)