rpcs3
rpcs3 copied to clipboard
RPCS3
[Feature Request] Clean up / compression
Please do not ask for help or report compatibility regressions here, use RPCS3 Discord server or forums instead.
Quick summary
Since games "install" is there a method of deduplication between the "disc" and the "install"? There seems to be no method of deduplication or compression used. If the game is already installed, is there any possibility to delete the disc and just have a basic runner, or does install have many disc dependencies?
Details
Running MSFT Compact XPRESS 8K, on RPCS3 w/ Siren Blood Cure installed makes the total reduce from 15.8 to 10.7gb. That's a 47% decrease in size on disk. Game is 11.1GB now 8.42GB. HDD0 4.42GB now 1.99GB. So deduplication and compression can greatly reduce storage usage. In other benchmarks, games that are compressed load faster [HDD to Memory is generally more of a bottleneck than CPU]. Xpress 8k is a very light compression and I wouldn't expect any CPU to bottleneck load times. Unless they were already bottlenecked by the emulation itself[In my testing of running the compressed game, "system" process which is used for decompression, showed no more than 0.6% CPU utilization on my 10900f.]
1. Please describe, what part of RPCS3 would be affected by your feature:
- Gameplay
- Patches
- Installation
2. Please tell us, why your feature is important to RPCS3.
3. Please attach screenshots of the feature implemented in other projects.
4. Please provide your system configuration:
Please include.
There's nothing to deduplicate, the game disc and install data are two separated things even if they may copy files from disc to install data. You can't modify the game disc, that's a big no-no preservation wise and can corrupt the game, you can't predict what the game does with installed data.
Compression is another thing that is possible.
using windows compact on most things does a good job but not all games are compressible
Why not using squashfs, batocera-linux have something like this on the way (HERE). I don't know if it will be efficient but at least it worth the try.
The best fast transparent compression is zstd, but ntfs/windows doesn't support transparent compression of it [but most modern linux fs do]. However RPCS3, could implement zstd support.
CHD support, which is the appropriate format you want here, is already being discussed on #4021
I like the idea of zSTD for all kind of internal stuff. Installs, save games etc. Would be nice if RPCS could compress these things during idle cycles.
I argue, that zSTD compressed archives are superior than CHD. CHD does not always guarantee lossless compression. It converts programmatically the input data and only safes the parts it/the programmers, knew of. With an archive, you guarantee lossless compression, since you don't touch the original iso/cue,bin etc. I have played with zSTD the last days and decompression on Android and other embedded devices is blazing fast up to level 19. After that, RAM might become an issue. Since it is so fast, even decompression before playing is not really an issue. On PC with a lot of RAM, you could even decompress the hole game direct to RAM.
"I like the idea of zSTD for all kind of internal stuff. Installs, save games etc. Would be nice if RPCS could compress these things during idle cycles." For Linux if the disk partition use BTRFS automatically does transparent compression faster and with better compression level than zstd. Windows has NTFS compression not being high compression. I had used RPCS3 in both Linux and Win ... I recommend use an Linux distro how Ubuntu or even other Ubuntu distros (have distro using less of 1 GB RAM and with optmized RAM usage for users with computer 4 or 6 GB RAM).
"I argue, that zSTD compressed archives are superior than CHD. CHD does not always guarantee lossless compression. It converts programmatically the input data and only safes the parts it/the programmers, knew of. With an archive, you guarantee lossless compression, since you don't touch the original iso/cue,bin etc." CHD not always use lossless compression ? In any way PS3 games files also are compressed.
"I have played with zSTD the last days and decompression on Android and other embedded devices is blazing fast up to level 19." Really is faster, but not extremely more fast than CHD. Linux use zstd and lzo-lre to compress zram. LZO-LRE is more fast than zstd.
"After that, RAM might become an issue. Since it is so fast, even decompression before playing is not really an issue. On PC with a lot of RAM, you could even decompress the hole game direct to RAM." You say zstd use much RAM when decompessing data ?
CHD good detail is all files are in one container thus secure.
Probably a bad idea. Compression often does not support fast-seekable data. Which games often need.
Well, yeah, but of course you don't randomly choose a compression format then? And this is like, what.. I think pretty much every emulator had the same kind of problem to address.
10 years ago pcsx2 started to work on the obvious solution of xz, and today I guess there's also zstd that could be considered.
Okay, it's doable. If someone can prove in actuallity the performance benefits here, that would be very helpful. Probably a good idea would be to actually test out if a disc game really loads much faster if HDD0 (dev_hdd0) is mounted on SSD rather on HDD on RPCS3, proving that disk storage performance is a real-world bottleneck of RPCS3 performance.
I guess that's an interesting question, but to be sure super lame 2.5" hdds exist (if any I really wonder if file fragmentation couldn't be something that can become big enough of a stumbling block to warrant some check?).
Though putting aside those cases where you are kinda asking for troubles, I think the greatest benefit of compression is lower storage requirements, not performance (at most you'd have to figure out the optimal block size, but I believe it should be pretty much a given that even if you had to recur to the smallest ones the file size improvements would still be significant).