scp-fs2open
XZ (and ZIP?) Support for compressed game assets
I wanted to comment of this idea before making a PR, when i made the LZ4 support last time it was a mistake to just PR it without a discussion first.
My idea was to add support for reading XZ (or ZSTD) files directly on FSO, and then propose to upload the files directly in this format to Nebula, instead of the 7z file that has to be extracted, uploading in this way you have a minimal impact in file sizes uploads to nebula, but would allow to use those files directly rather than having to extract them after upload. Also by keeping the copy of the file as it, rather than extract and delete it, diff patching for updates is a possibility, at least its viable, but that part is beyond of what i know how to do atm, so i cant promise that will be ever be done.
The two formats ive considered are ZSTD and XZ, ZSTD still allows for super fast decompression speeds nearly as the current LZ41 system can do. But i discarted the format before the frame system(needed for random access) is not mainlined in zstd and so no compressor supports it, so we need once again to make specialised compressors. Also my tests for file sizes werent that good, while considerable better than LZ41, LZMA2 can achive better results.
XZ (liblzma) was my second choice, and the one i started to test on. For file sizes, a full MVP 4.7.3 upload is about 350mb bigger than the .7z version (ZSTD is about 900-950mb).
I already did a test implementation into FSO for single files, but loading times are affected (as expected, LZMA2 is not as fast as LZ4). With basic multithreading support and some optimisations, i managed to get in a test system 1:07m loading time for the trailer mission, compared to 0:42m in a direct NVME with no compression. I havent profile it yet, so i hope this to improve a least a little.
Now im trying to figureout the workflow for containers and how i can apply that to what is already in. Im hoping to be able to support ".vp.xz" that could be later be extended to another container format, like ".tar" that could play along nice with .xz.
The big advantage of XZ, and maybe "tar" later too, is that the random access support is already built-in. I can compress .xz files with the 7z tool and use that directly on FSO, in fact thats how ive been doing my tests, no special compressor whatsoever. The only requerement for random access is to use small solid block sizes, in the area of 1MB-4MB.
I like the idea of .vp.xz as it avoids my biggest issue with the previous compressed VP situation (same extension and header being used for a different file format).
That said, I'm unsure how much gain we'd get from mutithreaded decompression. How are you expecting multithreaded decompression to affect load speed? Here's my current thoughts:
- The major bottleneck on data loading is that it's all sequential, the code reads one file after another, with no parallelisation of file accesses.
- This is fine for HDDs, but it's the main bottleneck for improving NVMe read speed (we want high queue depth).
- I don't expect to see major improvement to read speeds by multithreading the decoder if the mission loading code is still expecting to load/parse files sequentially, the latency of a random read isn't something we can improve with faster decoding.
- The main bulk of loading is POFs and textures, loading POFs in parallel might not be the easiest thing as they touch a bunch of game state, but the textures are just streaming disk data into RAM and then copying to the GPU.
Given that scenario, would it make more sense for N independent single-threaded file loader/decoders vs one multithreaded decoder?
It's a different problem to solve but it'd also improve load speed for raw files, vanilla, and compressed VP files too.
On diff patching:
- Building archives is easy, there's no real need to diff them in place
- The issue is there's no nice way in FSNebula to get a full list of folders and files (hashes) present in a VP file.
- This is something that should be fixed in infrastructure rather than in file format.
Yeah, file reads are intended to be secuential, in fact pack files (files inside a VP) they all share the same file pointer.
MT decompression is already in on my current tests. The effect is considerable, it went for +3m load time to 1.
From my tests there are two bottlenecks, one is model load, and the other is texture load.
-Model load is weird, it involves a lot of seeking and small reads (around 200 bytes each). Here you cant MT, the reads are just too small for it. But i used the same optimization i did for LZ4, keep the last decoded block in cache, so if model load request data from the same block sequentialy it can copy from cache rather than having to read the file and decode the block. This one has a massive impact in pof parsing.
-Texture load involves in reading the file header and then close it or load the whole file into ram. Here is where direct NVME has a massive advantage, this happens in the last part of mission load and where you can see disk read speeds reaching around 2000MB/s peak. The only thing that can keep up with that is LZ4 really.
For that i went for MT decompression, the general idea is:
-
First check how many blocks we need to decode, this is easy enoght to calculate. If we only need to decode one block to meet the requested data lenght, then it uses an optimised single block decoding method.
-
If we need to decode more than one block then all these blocks will be decoded in parallel. This makes a huge diference specially during texture load.
As for diff, last time we talked on discord, a few years ago, we arrived at the problem that knossos deletes the files that are stored on the nebula server after extraction. If instead of extract and delete the nebula files we where able to use them as they are, something like zsync could be implemented. But dont take my word for it, i have no knowledge in this area.
Another thing i wanted to talk about is possible support for other type of container format than VPs. My first idea was tar since it is commonly used with XZ but i completely forgot tar have no index and should not be used in this use case. I went on looking and the other strong condidate i see are zips. XZ is a supported compression format for zips, and so, supporting zip files in store-only mode and XZ (LZMA2) compression is possible. But the other compression modes would be unsupported. Maybe a warning in case a unsopported zip is detected is acceptable?
The good thing about containers is that there is no need to include additional libs for them.
So here is were im now in this whole idea:
- Upload ".vp.xz" to nebula, for an entire MVP upload thats around +450MB bigger upload than what is is now.
- In the daily use, using ".vp.xz" result in longer loading times, currently trailer mission is about 20-25s longer to load than direct nvme, (memory usage is not impacted in a meaningful way, if at all), this should not be that much an issue for regular missions as they dont general load more than half of the assets in one go, but it will be there.
- In Knossos side, that will result in mods install only needing to download the files, but for people who want to keep having the speed if they dont care about space or want to use VPCs thats a compromise of space and speed, Knossos would have an option to extract those files and keep doing things as of now, download->extract->recompress to VPC if wanted.
But ill need to check if old Knossos can do the extractions with the 7z command it uses, thats the one thing im unsure of here. I dont want to make old Knossos incompatible if i can avoid it.
- Singles files would not be compressed before upload at all. Single files (.xz or .lz41) are supported btw, its not that. Mod using unpacked files will still result in 7z LZMA2 compression and extraction during install.
- ZIP support might be added or not, its not hard at all to have zip/zip64 support i already made a simple parser whiout needing additional 3rd party libs and managed to implement it into FSO. But i havent commited this into the draft pr. The idea with ZIPs is not to use it now, but to have it as an option for future usage, specially since they dont have size limitations as VP have.
ZIPs support uncompressed data and the XZ format, so any zip created for use here, could be extracted by any standard zip software. To create a XZ compressed zip for FSO would need a compressor that support the zip format, allow you to select XZ method and 1MB dictionary and 1MB solid blocks for optimal speeds. Right now only 7z cmdline tools allow you do that, peazip do allow for XZ selection for zips but not blocksize. But ive been asking some people behind compressor programs if they could add it, since it is officially supported method after all.
As for deflate, the default method any zip program uses, there is no way to random access that, but i actually like the idea of maybe have a "compatibility support" for it so if you have deflate files into a zip, they would be completely decompressed into ram upon request, and free it when they are closed. This is very bad overall, but it would be ok for making some modifications. Like open a zip with winrar, extract a table, a model, a texture, mission, etc from it, modify it, and add it back. That individual file will be on deflate method while the others will still be on XZ, if the compressor dont support XZ. So having that would be convinient, as long it is just a very few files its not going to be a problem. And since FSO already uses Zlib for PNGs i think it can be used to extract deflate from zips too. But still, supporting deflate, even in a degraded way, involves some challenges, as they cant be detected in the same way as LZ41/XZ nor they contain the additional data to extract them, so it will involve passing some data from pack files loading (ZIP index contain the data needed) all the way to CFILE open for this particular case only. But, im not expending time on this particular thing, since i dont know if any of this will be ever be merged. If both XZ and ZIP are merged i might look into supporting deflate too at some point after the CFILE refactor.
In short:
.vp.xz -Allows to add an option to knossos to download and use the files as they are, or to extract them after download for maximum loading speed or to convert them to VPCs. -Allows users to extract them with any compression software for use in older versions -Limited to 2GB(uncompressed) forever
.zip -Support extraction and creation with any compression software that supports store mode or XZ mode, currently 7z support creating zip files with XZ mode and the desired configuration if you add the parameters -Initially limited to 2GB as there are still signed ints in CFILE, but this limitation would be removed after a CFILE rework -Support Zip64 mode (16 EB), this would need more testing after cfile rework -Loading is marginally faster that one big xz file for some reason, but nothing too big to consider it an advantage -Cant add support to extract them in knossos to get the full loading speed, as extracting a zip would be like extracting a vp and everything would be extracted into the same data folder potentially overwritting files. This is the major drawback compared to vp.xz, BUT maybe, just maybe i can to knossos the posibility to taking a XZ compressed zip and convert it into an STORE zip, achiving the same result. SO this is a maybe rather than a no.
So the question is: Support one or the other, or both? I see advantages in having both, but also having both might be seem as unnecessary.
I mentioned it in discord already, but repeating here,
With basic multithreading support and some optimisations, i managed to get in a test system 1:07m loading time for the trailer mission, compared to 0:42m in a direct NVME with no compression. I havent profile it yet, so i hope this to improve a least a little.
In the daily use, using ".vp.xz" result in longer loading times, currently trailer mission is about 20-25s longer to load than direct nvme
With the context of As a player this would be a serious annoyance to me, and as a modder it's worth screaming about. I can easily get (further) distracted from what I'm doing in that 20s, breaking my flow of play, and that's something I struggle with when playing mods already. And if I'm iterating on how mod tweaks feel in a mission that extra waiting might actually add up to a significant reduction in the ratio between productive time and waiting.
It's definitely a tradeoff that other people may feel differently about, but I'd take bigger mods with faster loads without hesitation. I'd be very reluctant to make it an optional thing too, because if as a user I found something tucked away in the options that made loads faster and it defaulted to off, I'd be cranky about it.
Hard drive space certainly isn't free, but I don't feel like for the user side the storage benefits are worth it. For the server storage side, and for transfer speed and people with bandwidth caps, it might be. Given that specific set of pros and cons, this feels to me like it should purely be a knossos feature, compressing the downloads and decompressing as an install step, without making the engine load the compressed files.
Well considering the lack of interest for a change to how nebula assets work and the fact this has a loading speed penalty, the different opinions and the amount of work still needed to be done (i never could get liblzma included in the cmake build proceess), specially on Knossos side, and i decided to retire the PR in order not to make things more difficult, specially with the upcoming SDL3 PR.
The work done here could be rebased and included at any time in the future, even if im not around anymore. You are allowed to use it.
This is my branch with XZ-only changes https://github.com/Shivansps/fs2open.github.com/tree/xz-support
And this for XZ + Zip support https://github.com/Shivansps/fs2open.github.com/tree/xz-and-zip-support
In both cases you still need to include liblzma to the cmake build process.
XZ support was done, it was a matter of seeing if anything could be done to speed it up more. ZIP was working fine, ZIP64 zips was only tested outside FSO due to internal changes were needed to use more than 2GB files. The only thing that was missing is a proper CRC32 verify on debug.