discord-api-docs
discord-api-docs copied to clipboard
discord
Cloudflare Workers unreliable due to Discord's Cloudflare bans (easy fix)
Description
Brought this issue up in #7137 and many agreed, including Cloudflare employees. though I think the Discord employee misunderstood me, he never replied after clarification given.
Experienced by myself and several others https://github.com/discord/discord-api-docs/issues/7040
Cloudflare Workers serverless uses Shared IPs to accomplish their massive scale. There are millions of developers sharing Cloudflare's IPs, and Discord's current configuration of their Cloudflare rate limiting rule makes using serverless unreliable with Discord.
Cloudflare Workers is a popular platform used by Discord bot developers. Cloudflare has a very active presence on Discord, assisting developers with Discord integrations on a daily basis.
Discord documentation recommended developers to use Cloudflare Workers numerous times, which makes this issue more surprising.
Cloudflare has a very strong IP reputation, and they do a great job at mitigating spam and DDoS through their platform. Discord wouldn't be using Cloudflare if it wasn't secure in this manner.
Fix
The IP address 2a06:98c0:3600::103 is used always for Worker requests to other Cloudflare websites (Discord is a Cloudflare website) https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#cf-connecting-ip-in-worker-subrequests (thank you @Skye-31)
Additionally, there is a cf-worker HTTP header which is only permitted use by Cloudflare workers. I tested sending header manually with my normal IP and a Cloudflare WARP IP, the header was stripped both times https://github.com/discord/discord-api-docs/issues/7137#issuecomment-2344767154
Steps to Reproduce
Send an API request to Discord from a high-load Cloudflare datacenter location. Here are the Cloudflare datacenters with problems so far:
- GRU (São Paulo, Brazil)
- CDG (Paris, France)
- FRA (Frankfurt, Germany)
- WAW (Warsaw, Poland)
The requested URL was https://discord.com/api/oauth2/token to exchange OAuth2 code
Expected Behavior
It would be expected that Discord API requests would succeed, with HTTP 200 code.
Current Behavior
Cloudflare Error 1015 (Global Rate Limit), when requesting the Discord API from Cloudflare datacenters that are high-load on Discord's API.
HTTP 429 code, because of Discord's configuration https://discord.com/developers/docs/topics/rate-limits#invalid-request-limit-aka-cloudflare-bans
Screenshots/Videos
Has occurred at least 1 time per week, for months. I can be sure our software isn't hitting these limits by itself, it's not that large. Even the total requests during that time period are under the limit, and Brazil is not even in our top locations for traffic to our software. It just so happens a lot of other Discord API developers have apps running in Brazil, it seems.
Client and System Information
Cloudflare Workers AS13335
IP address: 2a06:98c0:3600::103
our suggested approach to avoid IP bans from shared environments like Cloudflare Workers is to use a dedicated static egress IP. we will not be making Cloudflare Workers exempt from global rate limits.
Do note that is the wrong Cloudflare product and does not apply to workers. Closests would be Aegis, but that'd require a special orange-to-orange setup or Enterprise magic.
Needless to say, both products are enterprise only. If you have that much money, you can dedicate a server for proxying traffic instead.
I don't think there is a proper solution for this except special logic that replaces the IP address with the cf-worker header for global ratelimits.
please consult this documentation https://discord.com/developers/docs/topics/rate-limits#invalid-request-limit-aka-cloudflare-bans
our suggested approach to avoid IP bans from shared environments like Cloudflare Workers is to use a dedicated static egress IP. we will not be making Cloudflare Workers exempt from global rate limits.
Thank you for responding. It would be good if Discord looked into a separate, higher threshold for Cloudflare workers. Discord has the ability to add several different rate limiting rules on Cloudflare as an enterprise customer.
Also the cf-worker header is unique to each Workers website, so it's a good property to rate-limit by. That would prevent a few sites on Cloudflare workers getting spammed from affecting everyone on Cloudflare workers as a whole.
we are investigating using cf-worker as a rate limit key, but it may not be feasible for us. in the meantime a dedicated static IP is our best suggestion.
@afgiel any updates on this? many people are still trying to make discord bots using cloudflare, especially rookie developers who cannot afford the high costs of a dedicated server
Thanks to this thread I've finally realised what was going wrong with my new Discord bot on Cloudflare Workers!!
Surely this is something that individual small developers like me shouldn't have to resolve, the solution to use cf-worker as a rate limit key instead seems like a good one.
At the very least we should have better error reporting, I couldn't understand why my small bot was being rate limited!!
Was just directed here by a friend. Surprised that this isn't properly handled on a platform with as much traffic as Discord sees.
please consult this documentation discord.com/developers/docs/topics/rate-limits#invalid-request-limit-aka-cloudflare-bans
Large applications, especially those that can potentially make 10,000 requests per 10 minutes (a sustained 16 to 17 requests per second), should consider logging and tracking the rate of invalid requests to avoid reaching this hard limit.
What in this documentation should I consult? It's obviously not feasible for me personally to track the request rate from the IP my Worker or DO may be using.
Unless I misunderstood something, It seems like the only course of action is to just keep sending API requests and let them fail and wait until they eventually start working again. ... Or start micromanaging which Cloudflare servers my bot responds from?
Was just directed here by a friend. Surprised that this isn't properly handled on a platform with as much traffic as Discord sees.
please consult this documentation discord.com/developers/docs/topics/rate-limits#invalid-request-limit-aka-cloudflare-bans
Large applications, especially those that can potentially make 10,000 requests per 10 minutes (a sustained 16 to 17 requests per second), should consider logging and tracking the rate of invalid requests to avoid reaching this hard limit.
What in this documentation should I consult? It's obviously not feasible for me personally to track the request rate from the IP my Worker or DO may be using.
Unless I misunderstood something, It seems like the only course of action is to just keep sending API requests and let them fail and wait until they eventually start working again. ... Or start micromanaging which Cloudflare servers my bot responds from?
you just shouldn't use CF workers until this is actually done
15 requests a second is...sub 50 bots at ANY given time
We've worked with Cloudflare to resolve this issue so you shouldn't see incorrect 1015 errors while using Workers anymore!
