Discord bot becomes unable to remove roles from all users only after some extended period of uptime (~2-14 days)

[DeSinc]

Description

Only after some time of my bot being online and connected to the Discord API, when I request to remove a role with the RemoveRoleAsync(role) function in the Discord.Net library, it fails to remove any roles from any users. This only happens after some period of uptime. A reboot (reconnect to API) fixes this immediately for another 2-14 days or so before it happens again. This has been getting slightly more frequent over the last 6 months or so, and the result is the bot can add roles to people (such as the mute role) but cannot remove them, resulting in many permanently muted people to which the bot cannot remove the muted role it just applied.

Of particular note, one time I actually got an error 507: InsufficientStorage returned from the API when sending a role removal request. I didn't know this was possible. Where is the storage and how has it run out?

Error: Discord.Net.HttpException: The server responded with error 507: InsufficientStorage
   at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
   at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
   at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
   at Discord.API.DiscordRestApiClient.SendJsonAsync(String method, String endpoint, Object payload, BucketId bucketId, ClientBucketType clientBucket, RequestOptions options)
   at Discord.API.DiscordRestApiClient.ModifyGuildMemberAsync(UInt64 guildId, UInt64 userId, ModifyGuildMemberParams args, RequestOptions options)
   at Discord.Rest.UserHelper.ModifyAsync(IGuildUser user, BaseDiscordClient client, Action`1 func, RequestOptions options)
   at RoboSinc.Program.Tick(Object sender, ElapsedEventArgs e) in C:\Users\Dean\source\repos\robosinc-core\robosinc-core\Program.cs:line 440

Steps to Reproduce

Request a user's roles via the discord API (potentially specifically on my bot RoboSinc#7351) after some time of uptime has passed, say 2-3 days of uptime. It was working since the intents update but in the last 6 months or so it started to slowly fail and get worse.

Expected Behavior

When I ask for a user's roles, I should get a list of the user's roles returned to me by the discord API every time, and when I remove roles it should always remove the role even after 2+ weeks of uptime.

Current Behavior

When I request a user's roles, after some time of being up and connected to the API, I eventually get no roles returned (despite them having many roles) with no error message, implying a success, implying the user really does have 0 roles, when that is not the case. I cannot remove roles on any user anymore, but roles can still be added, resulting in many people getting the muted role but never being un-muted because the API seems to 'ignore' the remove roles request I send (RemoveRoleAsync(role)) with the Discord.Net library.

One time also receiving an Error 507: InsufficientStorage when trying to remove a user's role with either ModifyRolesAsync() or RemoveRoleAsync() using aforementioned library. The 507 could be unrelated but included it just in case.

Screenshots/Videos

My bot's intent config:

The code that attempts to remove the role (works for 2 weeks, then stops working randomly)

Client and System Information

Library: Discord.net 3.13.0 (C#)

[advaith1]

removing roles is a rest operation and has nothing to do with your bot's websocket gateway connection (which is the only thing affected by uptime). the issue you're describing sounds like an issue with your library's caching.

[Auralytical]

+1 to @advaith1's comment, I'm leaning on this being either a bug in the bot's logic or the library's caching.

As for the HTTP 507, as far as I know, we don't emit this status code from our API nor have I seen other users report this before. My best guess is this is being emitted from some layer between your bot and our servers (proxy server?). If you're seeing this regularly, it might be useful to inspect the response headers/content to get an idea of where this came from and why.

[DeSinc]

Thanks for both of your inputs. Gradually this issue is demystifying. The 507 explanation makes sense as it only happened once. I'm also checking with discord.net devs to see if they can identify a problem in their library.

[imptype]

I get this sometimes too?

<ClientResponse(https://discord.com/api/v10/interactions/1184495279228076082/aW50ZXJhY3Rpb246MTE4NDQ5NTI3OTIyODA3NjA4MjpQS2Z4SXo5Y2cwS2wwZVVvUDhtNlRGQnJYdk1NbHFhYTBNaGcxNlRtc1FPTWhZNHNiVmVsSmRmUzZEQkFONDdYWjIwT2hKTFNWQ3NIbHdBZkVoVEwwTzlEdGp0SjBZYlltaWgycTBsUkc1MDloWjBKeUtnQ0lyVGp6TnpUdHE0UQ/callback) [507 Insufficient Storage]>

<CIMultiDictProxy({'Date': 'Wed, 13 Dec 2023 14:01:21 GMT', 'Content-Type': 'text/plain', 'Content-Length': '53', 'Connection': 'keep-alive', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=86400', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=WosDilhtoXg1zcED9CfJ1hexKA%2FHH2GAasy66E4gPnGlW5%2FDQYr7Kivx8gKXhxNHOPydGWinLZHr%2BZKlAe3VXwYlAmqwuS%2BJj9%2BKcO8QWquYsyfCNrBe75D%2BCZVK"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'Set-Cookie': '__cfruid=f4b5e692e41050603955b162e3c9928e814a67af-1702476081; path=/; domain=.discord.com; HttpOnly; Secure; SameSite=None', 'Content-Security-Policy': "frame-ancestors 'none'; default-src 'none'", 'Set-Cookie': '_cfuvid=jbLCQmzhV8rvejdFDntdGqwf9Vvetqeb3CwTPG7NUWs-1702476081763-0-604800000; path=/; domain=.discord.com; HttpOnly; Secure; SameSite=None', 'Server': 'cloudflare', 'CF-RAY': '834ebd16bc95059e-IAD'})>

507 Insufficient Storage (error code: 0): exceeded request buffer limit while retrying upstream

It could be linked to https://www.cloudflarestatus.com/incidents/z07jsxj4vvzv

[DeSinc]

Changing how and when in the code that I called the SocketGuild has fixed this issue. Defining the socketguild at the start of the program and then using that falls over after somewhere between 1 and 14 days randomly (must have been a discord.net lib update that changed something) whereas defining a var for GetGuild every time I run the function now works fine indefinitely again. I am guessing it must be a lib quirk so closing this issue.