ppy
running out of memory when a lot of beatmaps are in the database
Hi :)
So, first of all I had the issue over a year ago already when I was at about 100k or 120k beatmaps. Meanwhile after certain updates of the client the Issue got better since it seems that the osu.db had been slim down a bit? The problems started to begin at around 140k or 150k beatmaps but I continued because I wanted to really stress test it. After all I was able to hunk in around 194k beatmaps(all ranked and loved maps) but the client became so unstable that it was almost impossible to play any longer. So what happens is that the client runs in 'out of memory' according to the logfiles. It happens most of the time in the songmenu. Typical bugs then are for example. 'could not load audiofile' or 'could not load beatmap'. Also often a background image simply wont be shown in the menu (the big backdrop, not the thumbnails). Sometimes the menu simply doesnt show any beatmaps at all. And in certain cases it even may crash either on song changing, start playing or after playing when trying to enter the results screen. And while the bugs occur, i get the message 'an error has occured in osu and has been reported' very often. Also it made the client crash when opening the options and scroll down there. If it helps, the bugs tend to occure more often, the longer a map is or probably, the more content it has to load, like storyboards or videos.
so, alot of issues which seem to be caused by one and the same problem. osu seem to run out of available memory it seems. Checking my Taskmanager, osu.exe was mostly consistant at around 1.5GB to 1.7GB of ram. My osu.db was at around 300 or 350MB.
for clearance, i run on 32 bit windows 7 in PAE mode with 16GB of RAM. Afaik even with PAE, the boundaries are still then 2GB of RAM per process. Like i said, taskmgr didnt show osu like exceeding it. however... I cannot tell how many processes osu runs or have more insight view as the sourcecode is closed.
To me it simply feels like when the database is around 350 MB, loading all of osu Interface media and then the preview of the selected song, it should never exceed 2GB at all or not?
compared to my current running (osu standard 3) folder I got in around 48k beatmaps and the database has ~100MB and osu runs smooth like a baby :)
I tested this around 2 or 3 weeks ago and since it was not playable any longer I split my collection of songs into 5 copies of the game. However if you need more information like the logfiles, let me know then I will reproduce the exact case.
tl;dr: too many beatmaps in the game (far beyond 100k) cause the songmenu to bug, even crash. logfiles point 'failed to allocate memory'
thanks for reading.
I am playing around in the instance of ~50k maps and watch the memory useage. <<client start, entering the songmenu it was: 506MB <<random song select 3 times, 560MB <<selected very short map 20 seconds, 569MB <<selected 32 minutes song, 581MB <<spamming RNG song select 620MB <<idle a while.... 900MB <<playing a song and re-enter song menu... some cleanup at least back to 712MB
while in songmenu now I am using systemInformer, select osu.exe->rightclick ->miscellaneous->Reduce working set the memory usage drops to 17 (!!!) MB and slowly increases ~1MB/s until i select a new song, it goes to >250MB
it's c# isnt? seems garbage collection isn't working proper? or isnt triggered often enough? how memory usage increases while idleing on the same song, no new resources should have to be even loaded. garbage collect seem to happen (partially working) when re-enter the UI after playing a map.
let me know if I can analyze and help any better.
This was improved recently by reducing the amount of things loaded from beatmaps. It should no longer be an issue with recent releases. If it is, you may have broken beatmaps loaded. Please provide your osu.db database for reference.
seems garbage collection isn't working proper?
that's not how it works.
also, your best bet is to use lazer instead, where this is a non-issue.
This was improved recently by reducing the amount of things loaded from beatmaps. It should no longer be an issue with recent releases. If it is, you may have broken beatmaps loaded. Please provide your osu.db database for reference.
seems garbage collection isn't working proper?
that's not how it works.
also, your best bet is to use lazer instead, where this is a non-issue.
Ok, it had happened on the last stable client already [20250401.2], cutting edge or beta was not different. Yes it had improved somewhen during the last year around that I noticed. I can be pretty sure that there was no corrupted import as I used the beatmap packs (zipped ones) from ppy package site directly and did fill up the missing ones using osu!direct.
If you want i can, as I said, go again and merge the currently 5 copies of osu into one with all beatmaps I have and after the diff calculation is done for every mode I can share the finsihed osu.db
However I clearly have doubts since the errors do not get like triggered onto the same maps, it is clearly random and a memory issue.
I know that switching to lazer would solve the problem but that would not solve the problem of stable. The description of this whole repository clearly points that it is for critical bugs and not like adding new features and in my opinion a client going nuts and even crash IS critical, so please at least TRY to solve it. If not at least confirm that I can debug it on my own and modify binaries when I am not logged in to bancho.
When existing problems aren't worked out together the whole issues repo for stable is pointless, when the 'fix' is to not use stable. besides that, peppy, I am sorry but maybe you misread that I can't even use lazer, since I am on win 7 32 bit.
oh and one more inportant thing. the db could not had be broken as it is still all the same songs. since i did split up the big one into 5 copies... all of them work... just not together... so I think there we go and you have the proove already
providing the combined database would be the next step in resolving this, so please do that if you can.
providing the combined database would be the next step in resolving this, so please do that if you can.
Thank you. Here are the 5 single osu.db files from the working-splitted copies of the game. Those are relevant as a reference before the merge. https://tasty.s-ul.eu/9VZqR4S1
now I will merge the song folders back together in a new copy of osu and will wait for the database to be fully written, that will take a good while, stay tuned :)
okay it seems not possible to merge back the full amount of songs at once because the client runs into - probably - a similar problem of not being able to allocate memory, the 'processing your beatmaps...' runs for a good while until the client starts to get completely not responding anymore. And after waiting, was away and come back it simply did close/crash. A osu.db was not even partially created so far. I tried it three times. I did attach the logfile showing clearly what is going on.
If it helps in any way. Here is a dump of the first hang: https://tasty.s-ul.eu/faJ9iqD4
I could bypass this step when able to merge my existing .db files to one together and place it manually. then I would F5 to scan again but the workload should then be minimal and not causing a crash.
Any possibilities? Like can you provide me a merged .db from your whatever using tools for .db analyzation? If not, can you provide the struct of how exactly the .db is made and I can write a small tool on my own then.
ok small update. Another attemp I did seem to work better but the issue still is critical. So first of all i did set the /3g switch on bcdedit to allow 3 GB per process instead of 2 GB. Also when started osu, I waited in the main menu until it did count all the ~190k maps and THEN went into play/solo which trigger the actual import. It crashed one time but on restarting the client it seem that it even resumed where it had left (detected ~40k maps). The importing went well and the calculation began. To not stress osu too badly I did close and reopen it like every 30 minutes when it had calculated a big amount of maps. It seem not to write the db directly but sort of caches stuff in /data/a ? Hence why I closed the client so the db gets updated and not again loosing parts on a crash.
As for now theres ~10k diff calcs to go for the mania gamemode. I will backup the written database and if needed can provide it here. When it's done I will also switch to the other 3 modes which will trigger again diff calcs for the corresponding modes and should increase the database again. Curious what happens this time :) However see the screenshot below that the client indeed hit the 2GB and would had crashed again if I would not had lifted the limits. So technically the Issue will happen even for 64 bit users sooner or later... guessing at around 320-350k maps =p
If theres anything else on how I can help to better analyze whats going on, tell me please.
Here you can see details about memory useage during the diffcalc https://github.com/user-attachments/assets/eb0449e6-5294-475c-a2bd-d2f537b3e124
Here you can see that the database seem probably not corrupted in any way, as my reader shows sane values until the end. https://github.com/user-attachments/assets/a6529acb-33d8-49ad-8bcf-bbf34546c3a5
So with the bcdedit /set IncreaseUserVa 3072 workaround osu didn't crash at the diffcalc but I had to set it back to default as it had too many side-effects =p