emmercm
Minor Cache Improvement - Cache the checksum info of ROMs in the output directory
Is your feature request related to a problem?
Short answer: Yes.
Long answer: A lot of the script setups I've seen and even what's suggested in the Maintainers Usage part of the guide, typically uses multiple calls to igir. First, it will incorporate any new ROMs into a "sorted / archival" collection, then it will copy subsets of the ROMs to various other locations (SD cards, external drives, etc.). My current setup is similar to this and its what a lot of users will go through as part of their first time setting up. This can be a pretty long process depending on how many ROMs are in the collection. I've had it take days when dealing with lots of games and large wii files.
Describe the solution you'd like
Something that is intended to help speed up subsequent calls to igir, is the cache! So given that igir saves all the metadata of those difficult to checksum ROMs in the cache, it would make perfect sense if it would save that data for the output directory as well. Now immediately after creating your "sorted / archive" folder, you'll get quick speeds copying over subsets of the collection.
Now perhaps this already how its intended to work, in that case we can discuss something similar to #1593 where this may be a bug instead of an enhancement.
Additional context
No response
I appreciate the suggestion. I've thought about this in the past and came to two points:
- It would incur a full file read after writing the file, to calculate the checksums of the file on disk and not what Igir thought was written (I sometimes come across test failures when writing files and haven't figured out why). And no time would ultimately be saved between two runs, it would just shift the cost of the full file read to the first command, which makes a lot of assumptions about what a user will do next.
- Or the result of checksums calculated during a file test could be cached, and maybe this is the right solution, but it might be hard to communicate that behavior to the user.
I'd love to hear if there's another strategy you've thought of that I haven't yet, I could have some blinders on.
I'm actually thinking more in terms of quick fixes or hacks, just to avoid having to re-analyze the entire collection every time I use it.
The craziest idea I’ve come up with so far is to use the dir2dat command and then compare the resulting DAT with the one that needs updating using SabreTools, so it only shows me the non-duplicates.
I haven't done any tests yet, nor do I know how it will turn out, but I can prepare the updates separately and implement them into the collection when I do the full check. Running a complete scan just to add maybe a single ROM is a waste of time, mainly because I’m still not familiar with the clean command, and I’m afraid of accidentally deleting something if I use it with only one DAT or some other incomplete data.
