schellingb
Load State Error: Invalid file format
Hi @schellingb,
Could you help with the "Load State Error: Invalid file format" error that occurs when calling Libretro’s retro_serialize() function? We’re using a custom Go frontend (which makes debugging the DOSBox core quite challenging), and it previously worked just fine with the DOSBox core—for example, with the December 20, 2024 version from RetroArch’s repository. However, I’ve noticed that state saving no longer works correctly in newer versions of the DOSBox core.
The issue is inconsistent. In Rogue, for instance, no error occurs in the main menu, but saving fails after starting a game. In MechWarrior 2, the error triggers immediately.
Would you happen to know if there have been any changes to state serialization that might explain this behavior?
Update. Found the reason: at our frontend, the save size was requested and cached only once, after the game loaded. In the case of Dosbox, it can change the size at any time, and now it apparently won't save the state with the old (larger) size. ¯(ツ)/¯
Thinking about this, the core should probably send RETRO_ENVIRONMENT_SET_SERIALIZATION_QUIRKS on startup to tell the frontend about this. These all would apply to DOSBox Pure:
/* Serialized state is incomplete in some way. Set if serialization is usable in typical end-user cases but should
* not be relied upon to implement frame-sensitive frontend features such as netplay or rerecording. */
#define RETRO_SERIALIZATION_QUIRK_INCOMPLETE (1 << 0)
/* The core must spend some time initializing before serialization is supported. retro_serialize() will initially
* fail; retro_unserialize() and retro_serialize_size() may or may not work correctly either. */
#define RETRO_SERIALIZATION_QUIRK_MUST_INITIALIZE (1 << 1)
/* Serialization size may change within a session. */
#define RETRO_SERIALIZATION_QUIRK_CORE_VARIABLE_SIZE (1 << 2)
/* Serialized state cannot be loaded on an architecture with a different
* endianness from the one it was saved on. */
#define RETRO_SERIALIZATION_QUIRK_ENDIAN_DEPENDENT (1 << 5)
/* Serialized state cannot be loaded on a different platform from the one it was saved on for
* reasons other than endianness, such as word size dependence */
#define RETRO_SERIALIZATION_QUIRK_PLATFORM_DEPENDENT (1 << 6)
There is also this part in libretro.h about the size function:
/* Returns the amount of data the implementation requires to serialize internal state (save states).
* Between calls to retro_load_game() and retro_unload_game(), the returned size is never allowed to
* be larger than a previous returned value, to ensure that the frontend can allocate a save state buffer once. */
RETRO_API size_t retro_serialize_size(void);
which seems to conflict with RETRO_SERIALIZATION_QUIRK_CORE_VARIABLE_SIZE.
DOSBox Pure actually does have the ability to return the maximum needed save size instead of just the size needed currently but that is only done if the "Save States Support" core option is set to "Enable save states with rewind". Thinking about it, it actually should always do that unless the frontend tells the core it has RETRO_SERIALIZATION_QUIRK_FRONT_VARIABLE_SIZE.
Will look into updating the core for that to make it more compatible.
As per API conventions, the core should return the maximum serialized state size when a ROM is loaded, and the frontend maybe allocates a buffer based on that (or whatever it wants to do). If the core uses variable-sized states, it can shrink the size later (then increase)—but it must never exceed the initial maximum until the ROM is unloaded. So no contradictions here.
Regarding quirks, it seems only a few non-mainstream cores use them (if you check GitHub sources). Honestly, it’s not a big deal—frontends can handle this with literally two extra lines of code. But hey, if you wanna do it "the proper way," why not? (:
Thanks for the response! I’ll leave the issue open if you need it for tracking, but feel free to close it otherwise.
The only reason the core works like this is because by default DOSBox runs with 16 MB RAM of which most of it is unused unless running a later era game. So having the core trim the size of the save data made sense to me 5 years ago.
But RetroArch of course by default compresses the save data written to disk. Most of the RAM bytes will be zeroes so even the most simple compression will be good enough so DOSBox Pure really doesn't need to care that much.
Can I ask how your frontend deals with the save state data? Does it use compression when writing it to disk?
I think it’s better to keep only the bare minimum of key functionality for the core (to minimize breaking changes). Everything else can live in higher-level layers.
The frontend in this app is built for multiplayer and cloud infrastructure, so compression is a must. What I do is copy a DOSBox Pure ROM filesystem snapshot as a base fs image into every new game session instance. I wonder if we should keep the base layer immutable and just let the diffs change over time. And the save states are optionally compressed with plain zip (skipped RetroArch’s weird rzip). The compression is optional because nowadays, tons of filesystems have built-in compression that works just as well.