undiscord
undiscord copied to clipboard
victornpb
"Ended because API returned an empty page" with 10K messages left
Undiscord Version
v5.2.0
Browser
111.0.5563.65 (Official Build) (64-bit)
Extension
TamperMonkey
Reproduction steps
- I initially had around 76K messages in a DM to delete
- I followed proper instructions, using TamperMonkey, and installed the script
- I ran the script, and bulk deleted around 66K messages in the span of 2-3 days
- This morning, I ran the script again with 10K messages remaining
Actual results
The actual result was that the window returned:
Fetching messages...
Grand total: 10565 (Messages in current page: 0 To be deleted: 0 Skipped: 0) offset: 87
Ended because API returned an empty page.
[End condition A]. {"running":true,"delCount":0,"failCount":0,"grandTotal":10565,"offset":87,"iterations":5,"_seachResponse":{"total_results":10565,"messages":[],"analytics_id":"2c2bbd338e4da352bd5b5fce114bd10f"},"_messagesToDelete":[],"_skippedMessages":[]}
Expected results
I had expected the continuation of deletion
Bug affects
- [X] DMs
- [ ] DMs (group)
- [ ] Channels
- [ ] Channel (NSFW)
Additional information & file uploads
None further
getting the same issue, undiscord currently broken
Ah damn. I thought it was an issue on just my end. Glad to know it's not just me
It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.
sorry and thanks for your understanding
It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.
sorry and thanks for your understanding
Hey, no worries! Thanks for everything you've done; it's been really convenient. Take your time, rest up, and make sure it isn't anything bad like a fever.
Take care!
It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.
sorry and thanks for your understanding
Is there a discord server for this script?
It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.
sorry and thanks for your understanding
How will we be informed that its been resolved?
expect updates on this thread
this issue will get closed, when I have a solution, and a new version will be pushed to everyone
expect updates on this thread
this issue will get closed, when I have a solution, and a new version will be pushed to everyone
im sorry for making a duplicate! i really didnt know there was one already that you actually responded to
Tried to use it after @hannahgrayson19 commented. Deleted 150+ messages, still going. Normally it would delete either 0 or 25 messages.
Update
The search results are no longer up-to-date, this is a optimization on discord side, as it makes sense to let the database slowly reindex deleted messages instead of make it instantly reindex, it's known that removing stuff on historical databases is a resource expensive operation. Unfortunately this makes things a bit harder on Undiscord side. The good news is that I already had a plan to allow Undiscord to prefetch messages. It's a bit of a happy coincidence but I still have to finish it, so this will be a more long term solution https://github.com/victornpb/undiscord/compare/prefetch. This will be in v5.3.0.
For now, I just released v5.2.1 (Make sure to click the update button, or just click the GreasyFork button again on the home page.)
I increased the defaultSearch delay to 30s between pages. Keep in mind this is a temporary workaround, and is not a complete fix, it still stops sometimes.
Error searching messages, API responded with status 401! {"message":"401: Unauthorized","code":0} CoreException {}
With the authorisation token filled:
Error searching messages, API responded with status 400!
{"code":50035,"errors":{"max_id":{"_errors":[{"code":"NUMBER_TYPE_COERCE","message":"Value \"[email protected]\" is not snowflake."}]}},"message":"Invalid Form Body"}
CoreException {}
Same problem
Error searching messages, API responded with status 400!
{"code":50035,"errors":{"max_id":{"_errors":[{"code":"NUMBER_TYPE_COERCE","message":"Значение «[email protected]» не является snowflake."}]}},"message":"Invalid Form Body"}
CoreException {}
Same here.
I can get a 200 removing &max_id= param
hope it helps...
thanks.
The search results are no longer up-to-date, this is a optimization on discord side, as it makes sense to let the database slowly reindex deleted messages instead of make it instantly reindex, it's known that removing stuff on historical databases is a resource expensive operation. Unfortunately this makes things a bit harder on Undiscord side.
@victornpb It'd be nice if we could use the data dump, that way it'd be able to avoid relying on the search endpoint entirely if it's available.
HI guys, just summing up to see if my understanding is correct on this issue:
1st - v5.20 has issues with the the deleted messages lingering on for ~1min and therefore not being able to progress due to ghost "blank page" which previously held the deleted messages.
2nd - v5.21 tries to solve that by implementing a quick-fix by increasing the page search to 30s (this most likely allows the re-indexing of discords' DB)
3rd - v5.21 has authentication issue: 401, etc., downgrading to v5.20 is the only solution to keep minimal functionality.
4th - v5.30 will mostly work-out these issues but it's more or less unclear when it will be released.
is there a current "quick fix" for the issues that can allow me to progress with deleting over 50k messages on a Discord server ?
@SorianoMarmol - is this a temporary fix ? if yes, how could I (with minimal technical knowledge) implement your proposed change ? (I can get a 200 removing &max_id= param)
I don't know how this works nor have I collaborated, I just know that I did a test removing that parameter and the response was 200. But as you can see in the screenshot, it can't even be interpreted, so I don't know if it's the expected response.
Regarding your question, and stressing that I don't know how this script works, maybe you can try removing the parameter from the call, I think it is this line:
https://github.com/victornpb/undiscord/blob/a9ce537bae6a40f343e09329848bd1caa394ccf0/deleteDiscordMessages.user.js#L658
Alternatively (outside of undiscord) you can always take the JSON that includes the messages (and their IDs) from the interface, parse the json to get the ids, and call delete in a loop with each of the ids. With Python it should be easy. For the base you simply copy the requests (and the json) from the chrome console (for example). Last night I tried to make call by hand changing the ID of the copied request and it worked. But doing it by hand is not feasible for 50k ;)
Regarding the part of 200 removing the param, it seems that it is irrelevant, today it already responds but it continues giving an error.
BTW: right now using undiscord with the token filled, the response is that indicated in the issue:
Ended because API returned an empty page.
Fetching messages... Grand total: 0 (Messages in current page: 0 To be deleted: 0 Skipped: 0) offset: 0 Delete delay: 1000ms, Search delay: 30000ms Last Ping: 236ms, Average Ping: 276ms Rate Limited: 0 times. Total time throttled: 0h 0m 0s. Estimated time remaining: 0h 0m 0s Ended because API returned an empty page. [End state] {"running":true,"delCount":0,"failCount":0,"grandTotal":0,"offset":0,"iterations":1,"_seachResponse":{"total_results":0,"messages":[],"analytics_id":"bdec65dd137eb5b2ee4756babcbb59a4"},"_messagesToDelete":[],"_skippedMessages":[]} Waiting 30.00s before next page... Ended at 30/3/2023, 19:38:10! Total time: 0h 0m 7s
{"fileName":"eval","lineNumber":1160,"columnNumber":16,"message":"JSON.parse: unexpected character at line 1 column 1 of the JSON data"} Could not automatically detect Authorization Token! Please make sure Undiscord is up to date Alternatively, you can try entering a Token manually in the "Advanced Settings" section.
It doesnt detect my auth token, not even when I manually add it :( (running 5.2.1)
Update! I tweaked around the delays in v5.20 and currently for me the following settings work:
search delay: 5000ms delete delay: 3000ms
I probably could go lower than 3000ms but the current settings give Discord enough time to re-index the search results. (From what I've seen it takes roughly 30 sec.)
It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance.
I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here.
It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again.
Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!
very very cool! worked.. and i found its working too with! 5000 2600
seems that you need to get here values where on one slide not more than 20-24 msgs get deleted
EDIT: stopped working but
5000 2800 is working perfectly for many K messages right now and still running
It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance.
I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here.
It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again.
Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!
see my post, just easy working with many K messages with setting 5000 2800 slow but working proper right now
It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance.
I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here.
It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again.
Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!
Not sure how to use your fork with tampermonkey
Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found.
https://pastebin.com/9Yq3r9Rw
Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.
Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found.
https://pastebin.com/9Yq3r9Rw
Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.
@victornpb Can you confirm if this works and push it into the next update?
Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found.
https://pastebin.com/9Yq3r9Rw
Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.
...could you open a PR?
It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance. I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here. It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again. Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!
Not sure how to use your fork with tampermonkey
You can find the compiled script here. The source code is also provided.
Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found. https://pastebin.com/9Yq3r9Rw Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.
...could you open a PR?
Do note that this page-blank fix opens the door to a myriad of issues. I knocked out quite a few of them in my fork, but there is one remaining bug that I did not test for accuracy. I would not recommend PRing just a page-blank fix as it will (probably) create more problems than it's worth.
I can review potential bugs a bit later and potentially PR that when I am reasonably confident no further bugs will propagate.
