Test suite for console-verified movies

Open RetroEdit opened this issue 5 years ago • 0 comments

With movies getting console-verified, it might be nice to create a test framework for ensuring cores continue to playback movies correctly.

There is a mountain of caveats here. For one, I'm specifically talking about movies where the input logs didn't need to be modified to work on console. Additionally, I am not suggesting a test failing would mean the core necessarily even has a problem; it would just be an indicator of something that might need to be looked into. If some behavior on console turns out to be nondeterministic and it just happened to work for the one tester, then it could obviously pose complications for that particular game.

A very basic and core-agnostic method of comparison would be to playback the movie and compare the framebuffer at the end. This approach is not without flaws; just because a movie's inputs were console-verified doesn't mean it rendered all of the graphics accurately in the emulator the original playback. Also, palettes would probably be a concern.

More sophisticated methods could take a full memory dump and compare, but this approach has even more subtle ways comparison can fail, and may be too complicated to be feasible.