flycast
flycast copied to clipboard
libretro
Flycast not extracting/loading zip compressed CDI
I'm not sure if this was implemented or not for Flycast, and if it was not I am also not sure if there has been discussion. I've looked briefly through closed issues and didn't find a match to this, and I am aware of issue #421 which is slightly different (I think).
I've been converting most of GDI images to CHDs because they save a ton of space, and they're lossless. Now, CDIs on the other hand cannot be converted into CHDs afaik so my idea was to zip/7zip the single file CDI images and work with that, but apparently this does not work with Flycast.
In a few other cores there is an automatic detection of the file inside the archive, the file is extracted temporarily and the game is loaded from the extracted file. In Flycast I've tried by directly loading the ZIP file which fails, and also tried the playlist method that works in other cores: "path": "archive.zip#packed_image.cdi"
I was expecting RA/Flycast to extract the packed_image.cdi from archive.zip. Was this feature discussed and/or decided upon?
Thanks.
In a few other cores there is an automatic detection of the file inside the archive, the file is extracted temporarily and the game is loaded from the extracted file
This behavior is disabled because of naomi/atomiswave, and there is no way to enable it only for dreamcast games afaik. The only way i can see around this would be what i mentioned in #421, where the dump is uncompressed into ram (meaning it would use a lot of ram when reading zipped dc dumps).
This behavior is disabled because of naomi/atomiswave, and there is no way to enable it only for dreamcast games afaik.
Naomi/Atomiswave roms are dumped into memory, so I see that what is trying to be avoided is to fill large quantities of RAM with GDI/CDI images which can be 10x the size of Naomi roms?
Also, Naomi/Atomiswave romset archives need to have a specific name, couldn't we just distinguish what is arcade and what is not based on that and abstain from unpacking to disk those that don't match the list and dump into memory those that do?
Furthermore, it is fairly easy to access and list the content of a zip/7z archive without extracting it, even if it's a few hundered MB long, and you could just check if there is a single CDI in it and decide based on that. On the playlist side alone it could work like other engines and explode the path at the # and have the information that yes, it is a zip archive, but inside it is a CDI that needs unpacking first: archive.zip#packed_image.cdi
As for streaming zips, I've skimmed through the #421 discussion, and yeah it would eat quite a bit of RAM, but I imagine that you already need a decent setup to emulate DC games, which might have that spare RAM, but maybe not. Anyway, extraction at this point would be good enough for me because from what I'm told there is no way to CDI->CHD, not even an intermediate process between both.
couldn't we just distinguish what is arcade and what is not
No we can't, preventing the archives to be uncompressed is done before anything else, in the block of code that defines what the core accept and what it doesn't, specifically this line : https://github.com/libretro/flycast/blob/0e0ecadeefe48e31fce963df3ad2e677090b37f2/core/libretro/libretro.cpp#L2219
If it's set to true, archives will never be uncompressed, if it's set to false archives will always be uncompressed (breaking every arcade games), it's done before RA send the file to the core.
I understand, thanks for this explanation.
Correct me if I'm wrong: RA knows which core to use for a particular file, it then reads the info->block_extract setting for that core before extracting or sending the file over and based on that parameter it decides to unpack or not beforehand. Is it a fair simplification?
Having to keep unpacked CDIs is crazy to me, and having to manually unpack them defeats the whole purpose; surely there must be a way to achieve this; but from what you explained thus far there isn't.
Is it a fair simplification?
Yes it is.
On a sidenote, you should avoid cdi dumps if possible, @flyinghead said several times that they were buggy (i don't know the full details about this).
you should avoid cdi dumps if possible, @flyinghead said several times that they were buggy
When it comes to emulation, CDI is not mandatory so I avoid it at all costs. Unfortunately, dumps aside, there is a whole modding, indie and homebrew scene that depends on the format. There are quite a few worthy games and hacks that exist and are CDI exclusive. That is why I began the issue in the first place, I am converting most GDIs to CHD (ramifications notwithstanding), but the same cannot be achieved with CDI, which will occupy at least 50% more space than if it was able to be compressed in one way or another.
Everyone I've asked so far told me that CDI to CHD (even with extra steps involved) is impossible, and so I'm resigned to teh idea that zlib/7z is my only alternative (be it extracted or dumped to memory), or build Flycast myself with that ioption to false, or to just decompress manually to disk and then... yeah, bummer.
Anyway, I'll leave this open for a while until someone closes it for me or shoots down any hope for the possibility.
Hello,
I have created initial work on supporting zipped CDI and GDI files see: https://gist.github.com/Langerz82/7925a54e75685b1f4aad17782616f207
Kind regards, Langerz.
EDIT: Changes updated to include zip support of GDI files!! :dancers:
Hello,
Could we get this implemented sometime soon? DC roms are massive and take up heaps of room uncompressed.