libretro
[Feature Request] Dynamic Save States
First and foremost consider this:
- Only RetroArch bugs should be filed here. Not core bugs or game bugs
- This is not a forum or a help section, this is strictly developer oriented
Description
Alright, last idea I'm posting here for a while. Very pie in the sky idea, and one i've considered tinkering with in rust via: https://www.retroreversing.com/CreateALibRetroFrontEndInRust.
Basically, it's sort of a less jarring, more performant, and more natural feeling alternative to rewind. I call it "Dynamic Save States" or dynamic checkpoints. The idea is that the game automatically makes savestates during moments it determines are "safe", rather than a set timeout (like every 5 minutes). Safety is determined by "not a lot of motion on screen" and "no controller inputs" for roughly a second.
"Not a lot of motion on screen" is based on the principle that libretro already handles entire frames of the core's video buffer in memory. And these buffers, I imagine (but don't know for sure) are relatively small at 320x240 at the smallest and 720 x480 at the largest in terms of images.
So the idea is to save a copy of the prior video frame in memory in addition to the current. If no controller input is taking place and prior video frame is roughly equal (like 70 or 80%) to the current video frame for sixty frames (that is one second), then create a save state. Some notification or millisecond countdown should occur when this happens and/or begins to happen, etc.
[Expected Result] The end result is that the illusion of checkpoints in modern games is created in just about any context, the user doesn't have to micro manage saves to undo mistakes, nor do they have to put such a heavy load on their system making MANY save states per second with rewind. This is of course, as with any quality of life or gameplay affecting option, optional.
Interesting idea. I'd be into checking out any proof of concept you get going :)
You just gave me the motivation I need to finish this heh: https://github.com/rishooty/rustretro-tutorial, I'm at step 19. Once the base is finished I'll get to trying it out!
Note that i thought about this more, and it should be used in conjunction with a small timeout, like 30 seconds since the last state or 1 minute, etc, so it isn't just constantly saving. Not quite sure how I would deal with pause screens, but that may be a necessary evil that promotes using the retroarch menu to really pause, not sure.
Edit: heh actually, i just realized that if it continues to be roughly the same after 60 frames it could just not save anymore until that changes, nice.
I think the "not a lot of motion on screen" would exclude games with automatic scrolling, such as Shmups.
The idle buttons detection sounds like it would work with the vast majority of games.
In any case though, how would you avoid saving on static Game Over screens that are also unskippable (so you have no reason to press buttons to skip)?
Honestly, my brain was on rpgs and platformers and schmups were not in mind but great point. This is sort of why I thought it would be an adjustable fuzzy match value, but that would have to be like 50% match possibly unusably low.
As for static screens I actually came up with an idea later on: only bother to begin saving if x number of different frames have passed since the last.
Weird sounding but I’ll explain.
basically, we have an adjustable counter for how many minutes worth of “progress” have occurred since the last save. Progress in this sense means sufficiently different (frames failing the fuzzy match check) frames have occurred, consecutively or not.
this allows the user to adjust the frequency of such a save without it being based on time and still feeling somewhat natural (unless that game over screen JUST HAPPENS to occur past the limit).
so, say the setting is 3 minutes. 3x60 = 180 seconds x 60 = 10,800 frames. When 3 minutes worth of motion have occurred since the last save, save and reset this counter back to zero.
As for still being able to get hit by bad luck… all I can think of is to check if the current image (not previous) is mostly the same color (that is r repeated bytes) and cease the process until this is not the case (many GO screens are one primary color with some text).
huh. Maybe we can do frames with registered inputs rather than frames with motion, but still count the frames like in my newest idea.
Sorry to double post but I feel I must separate this from my prior: what if we used images to only check if images are almost exactly the same (90%), and then we apply my frame counting to “frames with registered inputs” rather than “frames with motion”
That is, only use image comparison to check for such static game over screens to cease the process temporarily.
I feel this would be a better option.
you could always let people choose which mode produces a better result. That is, image-based or input-based (or some combination of the 2). As with any heuristic, it's very unlikely that any one approach will work perfectly all the time.
Sounds cool! Two quick notes:
- Instead of comparing whole frame buffers it probably makes sense to downscale the buffer to something tinier (maybe using the median color in each area) so you can do lots of comparisons quickly, or notice if the new picture is substantially different from old ones. This could help with scrolling games or games with full screen effects.
- Have you seen my crate retro-rs? https://crates.io/crates/retro-rs I haven’t updated it for a while but it works pretty well. It doesn’t support audio yet as far as I recall but I’d be happy to receive PRs.
I haven’t seen your crate! I like the way it’s organized. By the way, my initial fixing of that tutorial is done! I have cloned it into another where I’m going to replace minifb and rodio with macroquad, then begin my UI and save state experiments.
I’m beginning to lean more toward input only. When x number of frames with inputs have passed, make it eligible for the user to save without a few seconds of no input.
If god forbid they get to a game over screen, the idea is that there will be a bar or timer in the upper right hand corner. This timer is delayed for every input.
Cuz like isn’t it typical gamer habit to button mash through the game over screens anyway lol.
Sounds cool! Two quick notes:
- Instead of comparing whole frame buffers it probably makes sense to downscale the buffer to something tinier (maybe using the median color in each area) so you can do lots of comparisons quickly, or notice if the new picture is substantially different from old ones. This could help with scrolling games or games with full screen effects.
- Have you seen my crate retro-rs? https://crates.io/crates/retro-rs I haven’t updated it for a while but it works pretty well. It doesn’t support audio yet as far as I recall but I’d be happy to receive PRs.
Sorry to bump this, but I've come up with a significantly better theory. And even tested this among multiple genres for consistency: the same "comparing images" idea, but entire dumps of the in game memory.
My method:
- Open bizhawk, open a game, and open the hex editor
- Dump the entire WRAM to bin files both during crucial moments and about 2 or 3 relatively close to eachother (while nothing is happening, is paused, or moved the character a single frame, etc)
- Run the attached simple python script to compare the differences.
Also see attached for tests. It seems that with the sorta exception of Final Fantasy Mystic quest, most snes games have a less than 1% difference between similar frames, paused ones, or relatively inactive ones. Not sure how this would work for other systems, as I would need to do more testing.
The safest method would then be to either:
- queue up a save every x mins and only begin performing comparisons for "safe" autosaves after then until it finds one
- set "queued save" as a hotkey that can be bound on a common pause key, like start
- have it occur whenever the menu is open
For options 2 and 3, it would only perform the the check for safe frames when the button is pressed or RA menu is opened.
mario3.txt mysticquest.txt rtype3.txt scv4tests.txt ssf2.txt binCompare.py.txt