[Feature Request] Zstandard support (zip/7z/chd/zst)

[Sanaki]

With the recent additions of zstandard support to RomVault (zip, 7z, and chd), MAME (zip and chd, mamedev/mame#11827), and libchdr (chd, rtissera/libchdr#106), this is a prime time to add support to RetroArch for these formats as well. For anyone unaware, zstd provides high speed compression and decompression with respectable compression ratios (significantly improved over deflate on all counts, and competitive with lzma depending on compression level). It's also the standard compressor for Dolphin's RVZ format.

Support for chd will eventually be handled by updating the libchdr dependency, so that one's relatively simple. For zip, it exists in the official spec as method 93, though I'm uncertain how simple it is to hook that up to the zstd dependency added by libchdr's update. 7z support should be about the same as zip, though again, I'm uncertain how involved that would be.

Less straightforward is support for zst, which is a single-file compression format (similar to gz in that regard). It's unclear if there would be particular benefit to supporting bare zst files as single-file archives, or whether the difficulty of implementing that support would be prohibitive. I'd love to hear opinions on this point, beyond the obvious "why not?"

#4122 is a similar issue on the surface, but is centered around zpaq suipport. I felt it prudent to open a new issue considering.

EDIT: Did some quick testing by adding cdzs to the Silent Bomber chd. First is without it, second is with. Significant improvement in read speed, which has advantages for mobile devices with limited I/O speed.

chdr-benchmark "Silent Bomber (USA).chd"

libchdr benchmark tool....
Read 356937984 bytes in 9.787469 seconds
Rate is 34.779429 MB/s

chdr-benchmark test.chd

libchdr benchmark tool....
Read 356937984 bytes in 7.134529 seconds
Rate is 47.711992 MB/s

I ran a further test on DVD type chds with pure zstd, and the difference is frankly shocking.

Standard DVD (Jak and Daxter US)
CHD size:     887,124,057 bytes
Ratio:        60.7%
Read 1460535296 bytes in 28.192692 seconds
Rate is 49.405534 MB/s

ZSTD-only DVD (Jak and Daxter US)
CHD size:     913,880,480 bytes
Ratio:        62.6%
Read 1460535296 bytes in 8.581942 seconds
Rate is 162.303008 MB/s

[InquisitiveCoder]

Just wanted to express my interest in the CHD support in particular, since the MiSTer FPGA project updated its libchd version two weeks ago. Zstd is great for MiSTer since it's the only (de)compression algorithm fast enough to use x8 CD speed, so if RetroArch supported zstd CHDs I could use one set of CHDs for both platforms.

[saulfabregwiivc]

The new CHD ZStandard compression must be added to each emulator which supports/loads CHD, not RetroArch. RetroArch is just a front-end (correct me if i'm wrong).

[Sanaki]

Both are necessary for different reasons. RetroArch support is needed for achievements to work.

[LodanZark]

The current Retroarch nightly build was able to run CHD-ZSTD with the upstream flycast and ppsspp cores (might be the only cores to support that format so far).

[Immersion95]

I think the frontend support is needed in RA when handling with roms inside archives (SNES, N64, etc...). Roms as archive should be handled by the core (CHD, MAME's zip).

Flycast, ppsspp and NeoCD are the only cores that support zstd-CHDs at the moment.

[hizzlekizzle]

I believe the holdup on this is that it needs to basically happen all at once in RetroArch and the various libretro cores due to the way statically linked platforms work.

[rtissera]

Regarding CHD, https://github.com/rtissera/libchdr/pull/106 implements it.

[Ferroin]

Less straightforward is support for zst, which is a single-file compression format (similar to gz in that regard). It's unclear if there would be particular benefit to supporting bare zst files as single-file archives, or whether the difficulty of implementing that support would be prohibitive. I'd love to hear opinions on this point, beyond the obvious "why not?"

I cannot speak to the difficulty of implementing this, but as a user this is actually a feature I would like to see, though I will admit it will almost certainly be a rather niche thing. In terms of compression ratio, zstd falls somewhere between 7-zip and ZIP, usually tending to be closer to 7-zip than ZIP in my experience, but it does far far better than either at decompression speed, which translates to faster load times (which as the benchmarks done by the OP demonstrated, is a large part of why zstandard support in CHD files is such a major thing).

This definitely won’t matter for ROM files less than 1 MiB in size, but I can already see a measurable difference in load times on less powerful host systems for NDS and even some N64 ROMs when compressed with ZIP versus 7-zip, so I would expect to see a practical benefit on those systems to using Zstandard given that it has significantly better decompression performance than either on those same systems.

However, just like with most of the other ‘extended’ compression types supported by ZIP/7-zip, I have found that many tools that support these archive formats still don’t handle zstandard compression at all, so I personally see support for the standalone compression format as a much cleaner way to reap these benefits in RA than using a container format like ZIP/7-zip (also, you avoid some overhead by just using the standalone format instead of an archive format with zstandard compression if you’re doing a file-per-rom because you avoid all the extra metadata and framing of the archive, and that in and of itself can have a nontrivial savings if dealing with a large collection).

[nonstatic]

If I understand this correctly, each core will need to update libchdr to support this. I looked around and saw that the PUAE core did this some months ago.

It seems if people want this in a particular core, they can open an issue at that core's Github page.

It would be cool if zstd support is noted somewhere in a core's documentation, so that we don't have to guess if it's supported or not. afaik there's no comprehensive list about which cores support it and which don't. The most info is here in this thread at the moment.

[sonninnos]

Yes, most of the cores already do support it, and I've been using these missing ones that are not yet merged for a long time already:

The missing piece really is to also update the libretro-common version.