Support for GameBoy NSO Save files

[javierhimura]

It should be great to add support in PkHex to NSO save files for Gen 1 and 2 games.

Even if there is no official release of those games on NSO, it is possible to inject them into the APP.

The main difference between those saves files and any other GB save files is that NSO adds a 104 bytes Header at the start of the save file. This header also has an additional SHA1 Hash of 40 bytes. As far as I found, Gameboy NSO is the only console to add custom headers to the savefile. GBA and N64 don't have this difference with emulators and original hardware save files.

I have tried to make a Pull Request implementing this save format, but I am unable to replicate the NSO checksum. In this branch, there is my attempt which is capable to read and write the NSO savefile, but when I write the save, the Hash is not valid for the NSO App

https://github.com/javierhimura/PKHeX/tree/NSO_GameBoy

Documentation: There is a save file tool that implements the conversion between NSO and emulator formats. All the documentation I found about the Headers, and Checksum is from this tool and its source code.

https://savefileconverter.com/#/nintendo-switch-online

Here is the Gameboy sourcecode with the information about the Header

https://github.com/euan-forrester/save-file-converter/blob/main/frontend/src/save-formats/NintendoSwitchOnline/Gameboy.js

The hash algorithm is implemented here in JS, but I have been unable to port to C#. This JS algorithm writes a 40 bytes Hash but C# SHA1.HashData writes 20 bytes. I replicate JS code to generate the 20 bytes and then Encode it into ASCII to obtain 40 bytes, but still the output doesn't match the one expected in the NSO app https://github.com/euan-forrester/save-file-converter/blob/main/frontend/src/util/Hash.js

I include an NSO save file of Pokemon Red extracted with JKSV and its conversion to emulator format with savefileconverter.com tool save.zip

[javierhimura]

After some testing, I have been able to replicate the hashing and encoding used by Nintendo Switch Online by using UTF8 Encoding. With this, it works exactly as NSO expects...only with .NET Framework.

It seems that UTF8 encoding gives different results in NET Framework than NET Core, but NSO expects the legacy encoding https://github.com/dotnet/standard/issues/1679

EDIT: I confirmed the encoding works with NetFramework. I have edited the save with PkHex, and the save is not recognized by NSO, but then I have passed the edit save through the same encoding algorithm on a NET Framework program, and the save is recognized by NSO. But I am unable to replicate the legacy encoding on NET Core. The Nuget package UTF8Legacy mentioned on the issue does not give the same result as NET Framework

[kwsch]

Minimal repro of hashing:

static void WriteHash(ReadOnlySpan<byte> input, Span<byte> result)
{
    // Compute hash into the result span, then inflate in place.
    System.Security.Cryptography.SHA1.HashData(input, result);
    InflateAscii(result);
}

static void InflateAscii(Span<byte> data)
{
    for (int i = 19, j = data.Length - 2; i >= 0; i--, j -= 2)
    {
        byte b = data[i];
        data[j + 1] = GetHexChar(b & 0x0F);
        data[j] = GetHexChar(b >> 4);
    }

    static byte GetHexChar(int value) => (byte)(value < 10 ? value + '0' : value - 10 + 'a');
}

Results for your uploaded save (matches):

e42d7393e9f02b278df75e07c40347728ed891ac
65 34 32 64 37 33 39 33 65 39 66 30 32 62 32 37 38 64 66 37 35 65 30 37 63 34 30 33 34 37 37 32 38 65 64 38 39 31 61 63 

PKHeX.Core tries to stay away from emulator-specific header/footer handling, because I don't want to continually need to update support for every emulator's exotic variant.

The SaveFile class is a little brittle, in that it requires a byte[] for interacting with data, and separate header/footer byte[] via SaveFileMetadata. Ideally, this inner byte[] would be a Memory<byte> and we'd retain a reference to the format handler that can handle the format specific re-signing.

The SaveUtil class handles recognition of save types by first checking if it matches any raw format, then it checks each handler to see if it's recognized -> split & fetch via inner raw data.

Providing better support for wrapped savedata without hacky workarounds is a non-trivial rewrite, but it's something I eventually want to do.