space-wizards
Full server snapshot/reload support
Mega-PR that fixes #2406 Requires #6189
Overview
The main idea behind this PR is that if your content codebase will support full game saves, you can run a command that will save everything in Content.Server/data/Saves/[insert date and time here]/ folder, so you can shutdown the server, then turn server back on, load the save file, and have everything running like shutdown never happened.
Also it can be used for server rollbacks. For example, when someone griefs the whole game for everyone, an admin can just load an autosave to revert things as they were before the raid.
Technical Details
This PR adds GameSavesSystem EntitySystem, savegame and loadgame commands, and a CVar to enable/disable autosaving before server shutdown or just disable game saves on the server for everyone except the host.
GameSavesSystem just serializes all entities into 1 YAML file, and then compresses it using ZSTD to reduce the size quickly, or decompresses and loads the file back if loading.
So basically. it's just a wrapper for things that are already implemented in the engine.
Since RobustToolbox has ECS structure, if we save all entities as they were at some moment of time, and then spawn the back the same way, it should also continue working the same way. So instead of doing the path that Replays took by just writing every NetMessage into all files, we save only the state of the simulation itself.
That means all EntitySystems shouldn't have any data saved inside them (except for caches), since they are disposed on shutdown and will not get saved into the file without proper ECS implementation. So, some (all) codebases will have to fix their code before they can turn game saves on.
Task List
- [X] ~~Basic functionality for saving/loading all entities at once~~ Already done in #6189
- [x] Basic
GameSavesSystem - [x] Server-side CVar to control game saves functionality
- [ ] ZSTD compression and decompression
PRs to do next
Even after full game saves are supported, there's still a lot of work to do on the engine side, such as:
- EntitySerializer optimizations, it's too slow and because of that it makes implementing some features like dynamic autosaves impossible.
- Code analyzers that will throw warnings (or even errors) if you make your code 100% unsavable (example: using Timer.Spawn() in EntitySystem) also will be nice, but i think they need a separate -1 difficulty category tbh, they are indeed VERY hard to implement.
- Save migration support: Right now SS14 has some basic functionality for migrating EntProtoIDs, but that will definitely be not enough. Since the game is constantly updating, systems get refactored all the time, and game saves can become broken quickly. Some codebases may require some tools to migrate these saves, at least as C# scripts that are automatically executed before all saves are loaded to fix them. We can't implement migrations until we will have actual problem examples, that's why it's something we should worry about later and not now.
Impact on SS14
For saves to be actually supported at least in SS14, it will require a lot of content changes, since game saves will have no mercy on any code that violates ECS principles (and this has some potential to nuke downstreams).
When i successfully saved a Bagel station with Nukeops gamerule in SS14 using tools from this PR and content fixes, the resulting file weighted 16 megabytes and had tons of bloat. We have to improve custom serializers to have less bloat to reduce the file size and make it faster.
@ElectroJr Can I branch off the current RobustToolbox branch so I can apply this change on my Frontier server already?
Earlier, I had already made several attempts to implement this idea myself - and successfully a few times but I didn’t have the patience to finish the code, so I discarded it, since there are many other tasks to work on :)
Can I branch off the current RobustToolbox branch so I can apply this change on my Frontier server already?
Many components don't even save correctly on content are you sure it even works remotely correctly.
I'll rewrite that PR soon so it will be based on #6189, with a new file format for saves and ZSTD compression
I'll rewrite that PR soon so it will be based on #6189, with a new file format for saves and ZSTD compression
Are you okay with that one getting merged first then?
I'll rewrite that PR soon so it will be based on #6189, with a new file format for saves and ZSTD compression
Are you okay with that one getting merged first then?
Of course, it does maploader stuff better
I don't really understand how to use ZSTD, code in the PR is fully broken right now. Would appreciate any help, but anyway i will get back to this PR some time later eventually.
After ZSTD compression is propebly implemented it's almost ready, the only things left to implement are some special utilities like automatic naming for saves as a date-time, and automatic loading of the latest gamesave after server initialization.