unknownbrackets
Allow larger block sizes than 256 MB
This is a feature request and mostly a real world requirement in these days:
Please allow larger block sizes than 256 MB, up to 8 GB.
We are in the era of modern technology and competent PCs, where any advanced user should be able to specify a larger block size in order to (try)reduce the size of generated CSO file.
I have zero issues running CSO files compressed with maxcso using the maximum allowed 256 MB block size with zopfli, loading the CSO files in PCSX2 and PPSSPP. And my computer is from year 2017, it is not and it was not a high-end PC in any way.
Some PS2 ISO files from 'REDUMP' like "God of War (Europe, Australia) (En,Fr,De,Es,It).iso" are very huge, in this case 7,46 GB, and nothing -except 7zip- compares to the file size reduction that I can achieve using WinRAR (RAR5 format) with a dictionary size of 2GB, which generates a RAR file of 5,33 GB compared to the CSO file of 6,26 GB that maxcso generates using zopfli with 256 MB block size (by the way, it is the same resulting rounded file size as using zopfli with 128 MB in this case, for this ISO).
256 MB block size falls extremely short. Just please consider to allow the usage of larger block sizes (starting with 512MB, 1024MB, 2048MB and so on) for those who just want to test the compression results and who understand and accept the possible emulation issues on some very old computers and portable devices when increasing block size.
As an example, DolphinTool's CLI tool from Dolphin Emulator for GameCube and Wii, which serves to compress game files to RVZ format, it is even worse by warning all the time about emulation issues just for using 4MB block size and above (I never had any emulation issue on thousands of games GameCube and Wii games that I've tried with very larger block sizes), however it allows any larger block size assignation and the operation is completed as normally, it just warns about it but it allows the user to use whatever block size his memory slots can achieve...
Final note: I really don't understand how block size / dictionary size from WinRAR works in compression, I just proposed 8 GB as a limit since it is around the maximum file size I've seen for biggest PS2 ISO files. I know that a dictionary size of 6GB in WinRAR can profit / can make smaller file size when compressing a ~7 GB PS2 ISO file, but I'm not sure about 8GB dictionary size in this case, and in any case I'm not sure whether RAR's dictionary size is in fact the same as what we refer by the term 'block size' for other compression algorithms, but I think it should be very similar just different names.
Something to keep in mind is real hardware PSP compatibility, wouldn't be the first time (in general, not specifically this tool) hardware compatibility was broken from changes for emulation users.
I'm not sure if increasing the dictionary size would negatively affect hardware compatibility or not, or if there is an upper limit to it.
In addition to what @UltimateNova1203 said, why not use CHD for emulation? Modern PCs won't have issued decompressing LZMA in real time.
Isn't the point of block device compression primarily to permit random I/O to the compressed block device? Random I/O is only possible at the block boundary, that's the entire point for having blocks instead of compressing the entire stream in an equivalent of a solid archive. a 256 MiB block would mean that you'd need to decompress 256 MiB at a time every time you request a single byte (unless the entire decompressed block is cached, which will quickly degenerate into essentially keeping the entire decompressed ISO in memory).
Unless there's something I'm missing here, this is contrary to the very goal of block-based image compression.
Isn't the point of block device compression primarily to permit random I/O to the compressed block device? Random I/O is only possible at the block boundary, that's the entire point for having blocks instead of compressing the entire stream in an equivalent of a solid archive. a 256 MiB block would mean that you'd need to decompress 256 MiB at a time every time you request a single byte (unless the entire decompressed block is cached, which will quickly degenerate into essentially keeping the entire decompressed ISO in memory).
Unless there's something I'm missing here, this is contrary to the very goal of block-based image compression.
Hi. I don't know the technical details of how Random Data Access works in these compressed file type formats, but I can tell you that whenever I have the opportunity and the tool allows it, whether in CSO format with PSP, in GCZ format with DolphinTool for GameCube and Wii, or in NSZ format with Nintendo Switch, I use LARGE blocks, 128mb, 256MB, 512MB, etc., and the difference in performance is simply ZERO. It doesn't exist, at least in my case with my computer, which is from year 2017 with 16 GB of RAM and a outdated nVidia GTX 1060 so it's not a gaming beast in any way.
I've even measured it (rudimentarily using a digital stopwatch) using uncompressed ROMs/ISOs and ROMs compressed with these large block sizes, and we're talking about ISOs of several gigabytes. What I can tell you is that there isn't a single additional second in loading times (when booting the game and in any stage loading screens), no stuttering in emulators, no fps drops, or anything at all. Everything works exactly the same, with the added benefit of having the ROM/ISO with a higher compression ratio thanks to these block sizes.
The theoretical aspect of how random data access works will be one thing, and the practical aspect will be quite another. There is my testimony for the study. I understand that emulator developers will do their best to maximize reading efficiency, as long as the format in question allows it. And it must already be highly optimized by developers so that the impact on practice is absolutely zero or near zero (negligible effect).
After asking ChatGPT for his opinion, he told me two interesting things (although I'm not going to trust what an AI tells me, but as long as this can't be refuted, it's the only information I have and can comment on):
-
Using larger blocks can be beneficial, because using very small blocks, just a few kilobytes, generates a much larger table of offsets that can reduce performance efficiency.
-
Using larger blocks, as I do, may not have any practical impact because physically reading ROMs/ISOs is quite sequential, and ultra-random accesses to the ROM/ISO data never, or almost never, occurs. (I must say that ChatGPT gave me a PSP UMD disc as the only example)
Also I think emulators could cache read (decompressed) blocks so they don't have to decompress them again in memory, thus improving performance efficiency when a required offset needs to be read. This, combined with point 2, would further explain why there is absolutely no performance impact when using LARGE blocks.
But the reality is that I don't know, I'm just hypothesizing.
In any case, I can also tell you that in multi-gig ISOs compressed with large blocks, I have never seen the emulator process increase the private RAM to the size needed for either the total ISO size or even the block size. The explanation that ChatGPT gives me for this is that regardless of the block size used for compression, the data access operation works with much smaller buffers (e.g. 64 kb), and once the portion of data of interest is read, the buffer is discarded or overwritten in the next operation.
Sincerely, Elektro.