RetroPie
Slow initial startup shows static ES title screen for >3 minutes with no progress indicator.
I have many thousands of roms present, some of them without game list files. In this situation, ES takes forever to load on power up (~3 minutes). I think it is looping over all the roms to find the filenames. I don't understand why this is so slow, since it doesn't have to scrape or unzip the roms, and ES is written in C.
It would be nice to a) speed up the launch of ES on boot, b) display some kind of progress or at least a number in the bottom left corner, such as "6,125 roms found" with a number that increases.
From what I can tell, there were plans to fix this in upstream. Whether or not that's happening on the fork, no idea.
https://github.com/Aloshi/EmulationStation/issues/191
I added a comment in the upstream describing the problem and solution. It's not XML vs. SQLite.
@aarononeal upstream isn't maintained anymore, as far as I'm aware, but if you'd want to submit a PR here we'll gladly review it. Thanks!
EDIT: to be clear, I was referring to the changes in the file handling, as the progress handling you added there seems similar to the one we have here, unless I'm missing something.
The proposed fix mentions that the user should turn ON "Parse Gamelists Only". If I do that, my boot time is already very short. Of course it is short, because Emulationstation just parses a few XML files then.
My issue pertains to "Parse Gamelists Only" being OFF, which is the default option, and which is the only option that displays all my roms including subfolders correctly. I don't want to turn on "Parse Gamelists Only". I want Emulationstation to find and display new roms, and to display roms in sub folders correctly with subfolders that can be browsed as sub-menus. For that reason, I kept "Parse Gamelists Only" off.
I still don't understand why "Parse Gamelists Only" only should be required to be ON for fast boot time. Again: The app doesn't have to unzip or scrape roms, it just needs to go through the directories to find new rom files. That should take seconds, not minutes, even with 6000 roms. Reading a folder with 6000 file names and displaying the names takes less than a second in the shell.
Even my Vita with PFBA emulator can scan 13,000 roms in a few seconds on boot up of that emulator.
Of course it is short, because Emulationstation just parses a few XML files then.
That's not true. Without this fix, it checks the filesystem for every entry in the XML. This is extremely slow over high latency networks or media.
Ok maybe it is slow for you, but for me "Parse Gamelists Only" was always quite fast. Maybe 10-20 seconds boot time. But not minutes. The problem was when that setting was OFF.
The problem with the setting being ON is that only correctly scraped roms will be displayed. For example the Amiga emulator will not display any of my config files etc if that setting is ON, because well the scraper doesn't scrape them. I would still like to see them without editing xml files by hand.
There are 2 main paths, one parsing only game lists only and one scanning folders.
I optimized the parse only path by eliminating all stat calls to the filesystem. In my high latency test case, this had huge (160x faster) impact.
Looking at the other code path of scanning folders, I see stat is called for every entry in the gamelist multiple times. Depending on the media that's not as cheap as you might think. In some cases it results in updating last access time for each file. This may be contributing to the performance problems in that path as well.
I have some time to spare this week. Will take a look, unless any other maintainer is interested.
lstat is used mostly to follow symblinks for files. I am working on 2 approaches for this, either completely disable the symblink resolution flow using a param in settings or hard code the symblink resolution and work with the OS to auto follow the links. The second seems to be harder to figure out. Might take while so will keep posting status on this thread.