cdemu
Image MAME CHD - Initial scaffolding.
As per #16 introducing the feature requirement.
Was actually hoping to make more progress but for some reason as soon as you attempt to open the image it fades into the abyss and no debug information comes back.
It basically does the same as the benchmark test, which can be used to test a working image. We just want to retrieve the headers as part of the initial support for the format assertion...
err = chd_open(argv[1], CHD_OPEN_READ, NULL, &file);
if (err)
{
printf("\nchd_open() error: %s", chd_error_string(err));
return 0;
}
header = chd_get_header(file);
totalbytes = header->hunkbytes * header->totalhunks;
...which seems too much to ask for at this point.
Perhaps someone else can have more luck with this, hopefully it helps.
Attached find a simple test image of a disc with 1 text file which can be mounted with cdemu.
$ ls test.*
test.bin test.cue
The chd created with chdman createcd with the following output:
$ chdman createcd -i test.cue -o test.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.237 (mame0237-dirty)
Output CHD: test.chd
Input file: test.cue
Input tracks: 1
Input length: 01:36:05
Compression: cdlz (CD LZMA), cdzl (CD Deflate), cdfl (CD FLAC)
Logical size: 17,645,184
Compression complete ... final ratio = 46.1%
$ ls test.*
test.bin test.chd test.cue
Using the chd image on the libchdr benchmark test (working):
$ tests/chdr-benchmark test.chd
libchdr benchmark tool....
Read 17645184 bytes in 0.933225 seconds
Rate is 18.031835 MB/s
Can load cue file successfully with cdemu now to get the chd file to report header information...
Zip file attachment containing test.bin, test.cue, and test.chd: cdemu.zip
Ok success! The CHD-Parser now reports all debug messages as expected: (with the premature return FALSE; still set)
cdemu0: CHD-Parser: checking if parser can handle given image...
cdemu0: CHD-Parser: verifying image file's suffix...
cdemu0: CHD-Parser: opening image file for reading... CHD_OPEN_READ=1
cdemu0: CHD-Parser: parser cannot handle given image: invalid CHD file!
Yeah! \o/
I seem to have picked up another problem, the daemon crashes (segvault on libmirage) =(
This didn't happen with some of the previous iterations of the image plugin (on the 3rd). So either I removed something in the init or finalize blocks that should still be there, or I messed up something on D-Bus. Probably need to resolve this first...
Running on Ubuntu 21.10 (Impish Indri)
Message:
ERROR: Failed to load image: g-dbus-error-quark: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus withoutreplying (4)
Syslog reports:
Nov 9 19:44:36 boxname kernel: [78608.778631] scsi 7:0:0:0: CD-ROM CDEmu CD-ROM 1.0 PQ: 0 ANSI: 0
Nov 9 19:44:36 boxname kernel: [78608.778913] sr 7:0:0:0: [sr0] scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
Nov 9 19:44:36 boxname kernel: [78608.849151] sr 7:0:0:0: Attached scsi CD-ROM sr0
Nov 9 19:44:36 boxname kernel: [78608.849419] sr 7:0:0:0: Attached scsi generic sg2 type 5
Nov 9 19:44:40 boxname kernel: [78613.318555] cdemu-daemon[66732]: segfault at 4 ip 00007f1201866550 sp 00007ffe98b994f0 error 4 in libmirage.so.11.0.0[7f1201860000+1e000]
Nov 9 19:44:40 boxname kernel: [78613.318569] Code: 48 89 ce 48 89 c7 e8 4f bd ff ff 48 89 45 d0 48 8b 45 e0 48 89 c7 e8 af b5 ff ff 48 83 7d d0 00 0f 85 85 00 00 00 48 8b 45 c8 <8b> 40 04 83 f8 0b 75 12 48 8b 45 c8 48 89 c7 e8 6c c4 ff ff 83 45
Can find no additional information on daemon debug log.
How I may have messed up D-Bus, in the spirit of full disclosure:
While trying to find the cause for CHD-Parser fading into the abyss and none of it's debug messages appearing I noticed the folowing line added to the debug output:
_g_io_module_get_default: Found default implementation local (GLocalVfs) for ‘gio-vfs’
Thinking that there is some required package missing and going on a hunt to find anything gio-vfs related I could only find the package gvfs, which wasn't installed.
gvfs/impish 1.47.91-1ubuntu1 amd64
userspace virtual filesystem - GIO module
After installing the gvfs I got rewarded with an update to the intruding message:
_g_io_module_get_default: Found default implementation gvfs (GLocalVfs) for ‘gio-vfs’
But nothing else...
Having it installed or not doesn't seem to make any difference whatsoever, it is now removed again.
From google searches this message always seems to appear where there are dbus related issues, but it is never the topic of conversation so I am left to assume that it is yet another benign log message which was best left unreported unless specifically asked for. But that is beyond the scope of this exercise so returning to topic...
Odds are more likely that I messed up something in the plugin (unless mistaken and there actually is something amiss with gio?), so efforts will be going there instead.
Start the daemon in debugger, and then check the backtrace once it crashes. But for purposes of plugin development, it might be easier to work with image analyzer, or even with a simple C & libMirage program that just tries to load the image (much more straightforward to debug with gdb...)
Start the daemon in debugger, and then check the backtrace once it crashes. But for purposes of plugin development, it might be easier to work with image analyzer, or even with a simple C & libMirage program that just tries to load the image (much more straightforward to debug with
gdb...)
I can see nothing reported on the daemon debug output at the time of crash.
Could you assist with such a "simple C" program, not sure what is actually required to bootstrap a plugin. That would indeed greatly simplify the development if not required to start and stop the daemon and having to use cdemu-client to load an image hoping to see what is happening.
Update: returned the original unused class members and restored the init and finalize functions as they are for readcd (used as template) which had no effect.
Debug mirage log output:
cdemu0: CHD-Parser: checking if parser can handle given image...
cdemu0: CHD-Parser: verifying image file's suffix...
/home/intriazen/git/cdemu/test.chd
cdemu0: CHD-Parser: opening image file for reading... CHD_OPEN_READ=1
cdemu0: CHD-Parser: parser cannot handle given image: invalid CHD file!
ERROR: Failed to load image: g-dbus-error-quark: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus withoutreplying (4)
[1]+ Segmentation fault (core dumped) cdemu-daemon -n 1 --default-mirage-debug-mask=0x3ff
Syslog core dump info:
Nov 9 20:23:58 boxname kernel: [80968.590374] scsi 7:0:0:0: CD-ROM CDEmu CD-ROM 1.0 PQ: 0 ANSI: 0
Nov 9 20:23:58 boxname kernel: [80968.590517] sr 7:0:0:0: [sr0] scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
Nov 9 20:23:58 boxname kernel: [80968.664650] sr 7:0:0:0: Attached scsi CD-ROM sr0
Nov 9 20:23:58 boxname kernel: [80968.664874] sr 7:0:0:0: Attached scsi generic sg2 type 5
Nov 9 20:24:02 boxname kernel: [80972.639881] cdemu-daemon[68494]: segfault at 4 ip 00007fdc7981e550 sp 00007ffd294de380 error 4 in libmirage.so.11.0.0[7fdc79818000+1e000]
Nov 9 20:24:02 boxname kernel: [80972.639891] Code: 48 89 ce 48 89 c7 e8 4f bd ff ff 48 89 45 d0 48 8b 45 e0 48 89 c7 e8 af b5 ff ff 48 83 7d d0 00 0f 85 85 00 00 00 48 8b 45 c8 <8b> 40 04 83 f8 0b 75 12 48 8b 45 c8 48 89 c7 e8 6c c4 ff ff 83 45
Back to the drawing board...
Start the daemon in debugger, and then check the backtrace once it crashes. But for purposes of plugin development, it might be easier to work with image analyzer, or even with a simple C & libMirage program that just tries to load the image (much more straightforward to debug with
gdb...)
My bad I misunderstood, will try running the daemon in debugger.
Could you assist with such a "simple C" program, not sure what is actually required to bootstrap a plugin.
You just need to set up libMirage and try to load the image, e.g.:
/*
* gcc -o mirage_loader mirage_loader.c `pkg-config --cflags --libs libmirage`
*/
#include <mirage/mirage.h>
static void log_handler_stdout (const gchar *log_domain G_GNUC_UNUSED, GLogLevelFlags log_level G_GNUC_UNUSED, const gchar *message, gpointer unused_data G_GNUC_UNUSED)
{
g_print("%s", message);
}
int main (int argc, char **argv)
{
if (argc < 2) {
g_print("Usage: %s <image-file> [image-files...]\n", argv[0]);
return -1;
}
g_log_set_default_handler(log_handler_stdout, NULL);
/* Initialize libMirage */
GError *error = NULL;
if (!mirage_initialize(&error)) {
fprintf(stderr, "Failed to initialize libMirage: %s!\n", error->message);
g_clear_error(&error);
return -1;
}
/* Copy filenames from argv to a NULL-terminated array */
gchar **image_filenames = g_new0(gchar *, argc - 1 + 1);
for (int i = 1; i < argc; i++) {
image_filenames[i-1] = g_strdup(argv[i]);
}
/* Allocate context */
MirageContext *mirage_context = g_object_new(MIRAGE_TYPE_CONTEXT, NULL);
mirage_context_set_debug_mask(mirage_context, MIRAGE_DEBUG_PARSER | MIRAGE_DEBUG_IMAGE_ID);
mirage_context_set_debug_domain(mirage_context, "libMirage");
mirage_context_set_debug_name(mirage_context, "libMirage");
/* Load the image */
MirageDisc *disc = mirage_context_load_image(mirage_context, image_filenames, &error);
if (!disc) {
g_warning("Failed to load image: %s\n", error->message);
g_clear_error(&error);
} else {
g_print("Image loaded successfully!\n");
g_object_unref(disc);
}
/* Cleanup */
g_object_unref(mirage_context);
g_strfreev(image_filenames);
/* Shutdown libMirage */
mirage_shutdown(NULL);
}
Took some workspace re-configuring but managed to debug the daemon. Found the issue mentioned here:
This will pull the whole process into the abyss, because it doesn't set the error.
Setting the error first before prematurely returning FALSE, and it doesn't cause segfault anymore and it doesn't crash the daemon.
Removing the FALSE and again no debug info for CHD-Parser.
You just need to set up libMirage and try to load the image, e.g.:
Thanx stax for the assist. Doesn't help much cause I can't get cmake to build this for me.
Sigh!
You just need to set up libMirage and try to load the image, e.g.:
Thanx stax for the assist. Doesn't help much cause I can't get cmake to build this for me.
Sigh!
You don't need cmake to compile one-file program - just compile it manually using gcc command listed at the top of the source... (that's assuming that libmirage is installed in location visible to pkg-config)
You don't need
cmaketo compile one-file program - just compile it manually usinggcccommand listed at the top of the source... (that's assuming that libmirage is installed in location visible topkg-config)
Of course. Thanx that helps!
Ok mirage loader is build and works.
Still get no CHD-Parser output so it has probably got something to do with:
Do you have MIRAGE_DEBUG_IMAGE_ID enabled in context' debug mask? For the image I grabbed for testing, I can see:
Round two...fight!
Still get no CHD-Parser output
Does this mean you're missing only CHD-Parser output, or all debug output (for other parsers as well)?
We are able to open the file and get header information, the validation still fails at this point but deliberately, not due to something unknown... at least for now. =)
Still get no CHD-Parser output
Does this mean you're missing only CHD-Parser output, or all debug output (for other parsers as well)?
Found the culprit, after a long hard and persistent battle to restore order and sense back into the world.
Once I discovered the GTypeModule plugin parent class MiragePlugin the lights came on and there was a tunnel and a way out. Turns out pkg_check_modules with attribute REQUIRED (all in caps mind you) is only good for... well it's not any good actually, because what is the point of finding the library and then not linking it? Turns out we also need target_link_libraries to get rid of the broken symbols. That's how we learn...
So I guess the next step is to pass validation and see where it breaks? Any suggestions?
Thanx for the assist and patience!!
New output:
libMirage: CHD-Parser: opening image file for reading...
libMirage: CHD-Parser: reading chd header...
libMirage: CHD-Parser: MAME Compressed Hunks of Data (CHD)
libMirage: CHD-Parser: Input file : /home/intriazen/git/cdemu/test.chd
libMirage: CHD-Parser: File version : 5
libMirage: CHD-Parser: Logical size : 17645184 bytes
libMirage: CHD-Parser: Hunk Size : 19584 bytes
libMirage: CHD-Parser: Total Hunks : 901
libMirage: CHD-Parser: Unit Size : 2448 bytes
libMirage: CHD-Parser: Total Units : 7208
libMirage: CHD-Parser: Compression : Unknown
libMirage: CHD-Parser: CHD size : 8131772 bytes
libMirage: CHD-Parser: Ratio : 5
libMirage: CHD-Parser: MD5 : 00000000000000000000000000000000
libMirage: CHD-Parser: SHA1 : 8871799da800b02643e40b3afbf7e749aa32cc48
libMirage: CHD-Parser: Data SHA1 : 2b1eae918d6d59d540742b29f5120eaae3035807
libMirage: CHD-Parser: Metadata : 5
Still missing is Ratio, which looks like it needs to be calculated, Compression codec is not correct, and Metadata, for which method is yet undiscovered.
Need to make some time to get back to this.
Think the last hurdle I faced was how to list the contents....
Need to make some time to get back to this.
Please do. I've been wanting to convert my PSP collection from CSO to CHD for some time now but PPSSPP declined that PR for some silly reasons. Mountable CHDs would enable support for that right now.
Another interesting use case would be RPCS3, which doesn't even support ISOs. Usually the discs are encrypted but you can just decrypt the ISOs, mount them and play them just fine. That should also be possible with mounted CHDs. And although it has been said (again and again) that compressing game data from later console generations is pointless as the data has already been compressed, I can still observe some meaningful savings in e.g. The Last of Us:
| Format | Bytes | GiB | Size |
|---|---|---|---|
| ISO | 38636748800 | 35.983 | 100% |
| CHD | 35299023295 | 32.875 | 91.36% |
I know that the savings here are less than 10% but given the enormous size of PS3 games, it's still worth it in my opinion (10% here means more than 3 GB). The performance could be a concern though but I've been playing mounted ISO over SMB for a long time and I never encountered any issues. So please, don't abandon this :)
What about chd parents? For instance, in this dumb api python binding i made, from a library that doesn't actually mount anything, just parses the image into usable decompressed bytes, to support parent chd usefully we had to have a api to pass 'files' to the open call to try to recognize as parents, pretty much because each application will have a different usecase and the chd format never specified a 'standard'.
Could be 'chd in the same directory' could be 'chds in the same directory, or chds in the parent directory, or chds in any of the direct children directories' or something like that, but it would be good if delta chds were supported out of the box, without interface complication imo.
Btw, the library i depend on uses a alternative to libchdr in rust, which might be more readable or not, for comparing your implementation: https://github.com/SnowflakePowered/chd-rs
Update: some time has passed so the details are a bit foggy again.
Had issues trying to list the files inside a CHD to produce the table of contents.