Datacompression for rotations and and positions of belts

[mmjr-x]

This reduces the save game size of large saves by about 25% and since the Export and Import functions are also used for sending data over the network this will hopefully help a bit with the data transfer lag.

Some things to note:

• Sadly the optimization are not 100% lossless, however the rotational and positional differences are so minute that there is no noticeable difference (I also tried repeated loading and saving and noticed no degradation in precision).
• ~~I set the option to be disabled by default, so if you want to try this don't forget to activate it in the multiplayer options menu~~
• ~~If you enable this, any save you make will be stored in an optimized format. This format is not backwards compatible with vanilla which means that if for some reason you uninstall nebula, savegames made with compression on will not be loadable.~~
• Cross compatibility is maintained, so you can just load save games in the vanilla format and then save them in the compressed format if enabled. Or you can load an save game stored in the compressed format and then save it in the vanilla format if the setting is disabled.
• There are now 2 flavors for the compression options, a global one which is only available through the bepinex config which enables the compression for both normal saves and the network data transfer (this option is disabled by default). And network data compression, which only enables it for the data that is send over the network (this option can be changed through the in-game menu and is enabled by default)

Todos:

• Done, Move the compression code into its own non patch method so it can be used separately as well
• Done Possibly (or make it an option) make it so that compression is only used for data that is send across the network (this will ensure that people don't end up with broken savegame's if they ever decide to uninstall the mod).

Any suggestion for improvement is welcome, so review away! :)

[hubastard]

I think it would be better to make this as custom import/export methods that would only be used for the data transfer in multiplayer. That way we can never mess up the host save game. When the game gets updated.

[mmjr-x]

I think it would be better to make this as custom import/export methods that would only be used for the data transfer in multiplayer. That way we can never mess up the host save game. When the game gets updated.

I agree, and I will make the necessary changes for that, however if allowed I would like to keep an option (either in the settings, or in the bepinex config) that allows enabling it for normal savegame's too. The reasoning being that It is very hard to debug without having the option to enable this for normal saves :) . Please let me know if that would be acceptable?

[hubastard]

Ok it can be in the options but I don't think that we should display that settings in the game UI because players will try to turn that on to see what it does and it will probably mess up there save.

[mmjr-x]

@hubastard I have made the requested changes, however I think this requires some real world testing before it should be merged. Could someone who is versed at testing give it a go?

[mmjr-x]

Did some limited testing, fixed a pretty serious bug. I'm making this PR as ready for review, since I think it is good enough at this point (tried various other approaches for further positional compression, however all attempts gave problematic results). I might decide to continue trying other compression method's in the future, however that will come in a separate PR I think.

[PhantomGamers]

I'm getting this error on the host when a client tries to connect:

[Error : Unity Log] InvalidCastException: Specified cast is not valid. Stack trace: NebulaWorld.Factory.CustomExporters.CustomFactoryExport (System.IO.BinaryWriter w, PlanetFactory& planetFactory) (at C:/Users/Phantom/Documents/GitHub/nebula/NebulaWorld/Factory/CustomExporters.cs:39) NebulaNetwork.PacketProcessors.Planet.FactoryLoadRequestProcessor.ProcessPacket (NebulaModel.Packets.Planet.FactoryLoadRequest packet, NebulaModel.Networking.NebulaConnection conn) (at C:/Users/Phantom/Documents/GitHub/nebula/NebulaNetwork/PacketProcessors/Planet/FactoryLoadRequestProcessor.cs:25) NebulaModel.Networking.Serialization.NetPacketProcessor+<>c__DisplayClass34_0`2[T,TUserData].<SubscribeReusable>b__0 (NebulaModel.Networking.Serialization.NetDataReader reader, System.Object userData) (at C:/Users/Phantom/Documents/GitHub/nebula/NebulaModel/Networking/Serialization/NetPacketProcessor.cs:339) NebulaModel.Networking.Serialization.NetPacketProcessor.ReadPacket (NebulaModel.Networking.Serialization.NetDataReader reader, System.Object userData) (at C:/Users/Phantom/Documents/GitHub/nebula/NebulaModel/Networking/Serialization/NetPacketProcessor.cs:272) NebulaModel.Networking.Serialization.NetPacketProcessor.ProcessPacketQueue () (at C:/Users/Phantom/Documents/GitHub/nebula/NebulaModel/Networking/Serialization/NetPacketProcessor.cs:74) NebulaNetwork.MultiplayerHostSession.Update () (at C:/Users/Phantom/Documents/GitHub/nebula/NebulaNetwork/MultiplayerHostSession.cs:156)

Also I don't know if it's avoidable, but this breaks compatibility with CompressSave, which is pretty essential when you get to end game.

[mmjr-x]

@PhantomGamers The error should be fixed in eb8646a .

About compatibility with CompressSave I don't see why it would not work. I looked at the mod's code and it only seems to patch from the save and load from the menu's and autosave and what it does is exchange the memorystream for a LZ4CompressionStream. Since my compression works on the data level it changes nothing about the memory stream and it should just trickle down from the changes the CompressSave mod makes. The only possible thing that I could think of that would break compatibliliy is that the LZ4CompressionStream does not support setting position (which I do do as part of the version number detection mechanism). ~~I just installed CompressSave and and tried this in combination with the savegame compression of this branch, no conflict's and savegame's save and load fine with both compressions enabled :)~~ Scratch that probably if you try to load an uncompressed (by this branch) savegame (with the CompressSave on) you will also get the error that is in the comment below.

On a side note, I don't see how the CompressSave mod would work for the data send over the network at all? since it only patches the save menu's and nebula's FactoryLoadRequestProcessor.cs sets up its own stream. Or am I missing something?

[mmjr-x]

Ah crap, seems the stream that is used for sending networked data does not support seeking :( (Got this on the client when trying to connect)

[Error  : Unity Log] System.NotSupportedException: Stream does not support seeking.
  at System.IO.BufferedStream.EnsureCanSeek () [0x0000d] in <2fa7a6a452ca43df998f07fd1486c0df>:0
  at System.IO.BufferedStream.get_Position () [0x00006] in <2fa7a6a452ca43df998f07fd1486c0df>:0
  at NebulaPatcher.Patches.Dynamic.CargoPath_Patch.Import_Prefix (CargoPath& __instance, System.Int32& ___bufferLength, System.Int32& ___chunkCount, System.Int32& ___updateLen, System.IO.BinaryReader r) [0x00008] in <2205d6669a0545b3b3ad4ab5daa020d1>:0
  at CargoPath.Import (System.IO.BinaryReader r) [0x00005] in <8feb1a2572364e12966241a5fffeb105>:0
  at CargoTraffic.Import (System.IO.BinaryReader r) [0x00127] in <8feb1a2572364e12966241a5fffeb105>:0
  at PlanetFactory.Import (System.Int32 _index, GameData _gameData, System.IO.BinaryReader r) [0x00757] in <8feb1a2572364e12966241a5fffeb105>:0
  at NebulaPatcher.Patches.Dynamic.GameData_Patch.GetOrCreateFactory_Prefix (GameData __instance, PlanetFactory __result, PlanetData planet) [0x000ad] in <2205d6669a0545b3b3ad4ab5daa020d1>:0
  at GameData.GetOrCreateFactory (PlanetData planet) [0x00005] in <8feb1a2572364e12966241a5fffeb105>:0
  at PlanetModelingManager.LoadingPlanetFactoryMain (PlanetData planet) [0x0002c] in <8feb1a2572364e12966241a5fffeb105>:0
  at PlanetModelingManager.LoadingPlanetFactoryCoroutine () [0x00053] in <8feb1a2572364e12966241a5fffeb105>:0

Seems the internally used LZ4Stream does not support Seek sadly. Marking this PR back as draft until I found a solution.

[mmjr-x]

Should be good to go now, could someone test multiplayer for me? I sadly can't use the sandboxie plus approach to run multiple instances of the game to test myself anymore since it messes up my VPN networking config :L (I'll try and find a workaround)

[PhantomGamers]

I sadly can't use the sandboxie plus approach to run multiple instances of the game to test myself anymore since it messes up my VPN networking config :L (I'll try and find a workaround)

you can bypass the need to use Sandboxie by removing the single-instance= line from DSPGAME_DATA\boot.config

[mmjr-x]

I sadly can't use the sandboxie plus approach to run multiple instances of the game to test myself anymore since it messes up my VPN networking config :L (I'll try and find a workaround)

you can bypass the need to use Sandboxie by removing the single-instance= line from DSPGAME_DATA\boot.config

Thats so nice! :D Will try that immediately! (might be also good to add that to the dev wiki? :) )

[mmjr-x]

~~Mmm not getting it to work, the second instance immediately closes after boot (does not even get tot the main menu)~~ Working now, had to also put a steam_appid.txt into the game's root folder :) Thanks for the help @PhantomGamers !

[mmjr-x]

I just tested multiplayer, client is able to connect and it works in conjunction with the CompressSave mod as well. So I guess this is good to go :)

[mmjr-x]

I'm putting this back to draft, with all the updates and changes I'm fairly sure this needs some further love :)