emmercm
WARN: <chd_path>: found no files in path
Paste the command
igir report --dat /dev/shm/mame.xml --input ~/mame/roms --input-checksum-quick --merge-roms merged
using the XML file as input generated from mame -listxml in version 0.278.
Describe the bug
I'm seeing this warning for a fair number of my CHD's:
WARN: <chd_path>: found no files in path
It looks to be around 405 of them in total (about half of the CHD's I have).
Expected behavior
I'm not even clear on what the warning is trying to say. As far as I know, these are all good CHD images as clrmame has always verified successfully. Note that this is happening during the initial ROM scan, and not during the ROM trimming detection I just opened a separate issue about.
I thought it might be related to CHD version or some such, but it seems to indicate the same warning for both version 4 and 5 CHD files, so that isn't the common factor.
Debug logs
Right from the top:
[14:32:37.257] TRACE: ROMScanner: scanning ROM files
[14:32:37.396] TRACE: ROMScanner: found 16,229 ROM files
[14:32:40.606] WARN: ROMScanner: /home/nipsy/mame/roms/atronic/atronic.chd: found no files in path
There is no additional output beyond the warning.
DAT(s) used
No response
igir version
4.1.1
Node.js version
22.17.0
Operating system
NixOS
Additional context
No response
I'm not even clear on what the warning is trying to say.
It's trying to warn the user that it couldn't find any files within the archive, which should be an extremely unlikely scenario and worth notifying about.
Can you attach the output of chdman info --input <file.chd> of a few of those files? That will be required for debugging.
Here's just a random selection of some:
% chdman info -i atronic/atronic.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.278 (unknown)
Input file: atronic/atronic.chd
File Version: 5
Logical size: 9,792,000 bytes
Hunk Size: 9,792 bytes
Total Hunks: 1,000
Unit Size: 2,448 bytes
Total Units: 4,000
Compression: cdlz (CD LZMA), cdzl (CD Deflate), cdfl (CD FLAC)
CHD size: 7,095,515 bytes
Ratio: 72.5%
SHA1: 3335e9f8f67f1b176e043f078456d2b13178b7ef
Data SHA1: c463c3d54524a3ad52acef9d0744ed029f3aa291
Metadata: Tag='CHT2' Index=0 Length=85 bytes
TRACK:1 TYPE:MODE1 SUBTYPE:NONE FRAMES:4000 PREGAP:0 PGTYPE:
% chdman info -i cndypuzl/cndypuzl.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.278 (unknown)
Input file: cndypuzl/cndypuzl.chd
File Version: 4
Logical size: 434,431,872 bytes
Hunk Size: 9,792 bytes
Total Hunks: 44,366
Unit Size: 2,448 bytes
Total Units: 177,464
Compression: zlib (Deflate)
CHD size: 15,868,976 bytes
Ratio: 3.7%
SHA1: 5f41ed3521b3e05d233ac1245b78cb0b118b2b90
Data SHA1: 21093753a1875dc4fb97f23232ed3d8776b48c06
Metadata: Tag='CHTR' Index=0 Length=46 bytes
TRACK:1 TYPE:MODE1 SUBTYPE:NONE FRAMES:163384.
Metadata: Tag='CHTR' Index=1 Length=44 bytes
TRACK:2 TYPE:AUDIO SUBTYPE:NONE FRAMES:8027.
Metadata: Tag='CHTR' Index=2 Length=44 bytes
TRACK:3 TYPE:AUDIO SUBTYPE:NONE FRAMES:3354.
Metadata: Tag='CHTR' Index=3 Length=44 bytes
TRACK:4 TYPE:AUDIO SUBTYPE:NONE FRAMES:2694.
% chdman info -i jppyex98/811jaa02.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.278 (unknown)
Input file: jppyex98/811jaa02.chd
File Version: 5
Logical size: 31,226,688 bytes
Hunk Size: 19,584 bytes
Total Hunks: 1,595
Unit Size: 2,448 bytes
Total Units: 12,756
Compression: cdlz (CD LZMA), cdzl (CD Deflate), cdfl (CD FLAC)
CHD size: 15,231,258 bytes
Ratio: 48.8%
SHA1: e9580172d58b38841f643651ae0bcaf24fd5f118
Data SHA1: 2fc0b986936cc1a4350140c4d2c81b500230ff96
Metadata: Tag='CHT2' Index=0 Length=92 bytes
TRACK:1 TYPE:MODE1_RAW SUBTYPE:NONE FRAMES:11514 PREGAP:0 PG
Metadata: Tag='CHT2' Index=1 Length=89 bytes
TRACK:2 TYPE:AUDIO SUBTYPE:NONE FRAMES:1238 PREGAP:150 PGTYP
% chdman info -i quizard3/quizard32.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.278 (unknown)
Input file: quizard3/quizard32.chd
File Version: 4
Logical size: 475,352,640 bytes
Hunk Size: 9,792 bytes
Total Hunks: 48,545
Unit Size: 2,448 bytes
Total Units: 194,180
Compression: zlib (Deflate)
CHD size: 226,190,757 bytes
Ratio: 47.6%
SHA1: 31e9fa2169aa44d799c37170b238134ab738e1a1
Data SHA1: a676d881f1f0fccb154b81774c6edd6e8bb2b66c
Metadata: Tag='CHTR' Index=0 Length=50 bytes
TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:194177.
% chdman info -i quizard3/quizard34.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.278 (unknown)
Input file: quizard3/quizard34.chd
File Version: 4
Logical size: 483,431,040 bytes
Hunk Size: 9,792 bytes
Total Hunks: 49,370
Unit Size: 2,448 bytes
Total Units: 197,480
Compression: zlib (Deflate)
CHD size: 228,546,494 bytes
Ratio: 47.3%
SHA1: 37ad49b72b5175afbb87141d57bc8604347fe032
Data SHA1: 9acc5ee1883d27b2a500d139f7837c01936e6c77
Metadata: Tag='CHTR' Index=0 Length=50 bytes
TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:197477.
I've tested all of these and they run fine.
If it helps to see the entire list of all these possibly unrecognized metadata types, I can dump it all into a gist or some such.
Very strange, I tested a handful of these on macOS and they seem to be alright.
Can you try running Igir with --reader-threads 1 --disable-cache? I'm wondering if there's something wrong with multiple copies of chdman running at once.
Aside from being catastrophically slower of course, I'm still getting the same warnings:
% igir report --dat /dev/shm/mame.xml --input ~/mame/roms --input-checksum-quick --temp-dir /dev/shm --merge-roms merged --reader-threads 1 --disable-cache -vvv
@@@@@@ @@@@@@ @@@@@@ @@@@@@@@
@@ @@ @@ @@ @@ @@
@@ @@ @@ @@ @@ @@ ROM collection manager
@@@@@@ @@ @@@@@@ @@@@@@@@@ https://igir.io/
@@@ @@@@ @@@
@@ @@ @@ @@ @@ @@ v4.1.1
@@ @@ @@ @@ @@ @@
@@ @@@@@@@@@@@@ @@ @@ @@
[19:06:01.313] TRACE: Parsing CLI arguments: report --dat /dev/shm/mame.xml --input /home/nipsy/mame/roms --input-checksum-quick --temp-dir /dev/shm --merge-roms merged --reader-threads 1 --disable-cache -vvv
[19:06:01.314] TRACE: Parsed CLI options: {"_":["report"],"input":["/home/nipsy/mame/roms"],"inputExclude":[],"inputChecksumQuick":true,"inputChecksumMin":"CRC32","inputChecksumArchives":"auto","dat":["/dev/shm/mame.xml"],"datExclude":[],"datCombine":false,"datIgnoreParentClone":false,"patch":[],"patchExclude":[],"dirMirror":false,"dirDatMirror":false,"dirDatName
":false,"dirDatDescription":false,"dirLetter":false,"dirLetterCount":1,"dirLetterLimit":0,"dirLetterGroup":false,"dirGameSubdir":"multiple","fixExtension":"auto","overwrite":false,"overwriteInvalid":false,"moveDeleteDirs":"auto","cleanExclude":[],"cleanDryRun":false,"zipFormat":"torrentzip","zipExclude":"","zipDatName":false,"linkMode":"hardlink","symlinkRelative
":false,"mergeRoms":"merged","mergeDiscs":false,"excludeDisks":false,"allowExcessSets":false,"allowIncompleteSets":false,"filterLanguage":[],"filterRegion":[],"noBios":false,"onlyBios":false,"noDevice":false,"onlyDevice":false,"noUnlicensed":false,"onlyUnlicensed":false,"onlyRetail":false,"noDebug":false,"onlyDebug":false,"noDemo":false,"onlyDemo":false,"noBeta":
false,"onlyBeta":false,"noSample":false,"onlySample":false,"noPrototype":false,"onlyPrototype":false,"noProgram":false,"onlyProgram":false,"noAftermarket":false,"onlyAftermarket":false,"noHomebrew":false,"onlyHomebrew":false,"noUnverified":false,"onlyUnverified":false,"noBad":false,"onlyBad":false,"single":false,"preferVerified":false,"preferGood":false,"preferLa
nguage":[],"preferRegion":[],"preferRetail":false,"preferParent":false,"playlistExtensions":[".cue",".gdi",".mdf",".chd"],"reportOutput":"./igir_%YYYY-%MM-%DDT%HH:%mm:%ss.csv","datThreads":2,"readerThreads":1,"writerThreads":4,"writeRetry":2,"tempDir":"/dev/shm","disableCache":true,"verbose":3,"help":false,"d":["/dev/shm/mame.xml"],"i":["/home/nipsy/mame/roms"],"
input-checksum-quick":true,"temp-dir":"/dev/shm","merge-roms":"merged","reader-threads":1,"disable-cache":true,"v":3,"input-checksum-min":"CRC32","input-checksum-archives":"auto","dir-letter-count":1,"dir-game-subdir":"multiple","fix-extension":"auto","move-delete-dirs":"auto","zip-format":"torrentzip","link-mode":"hardlink","playlist-extensions":[".cue",".gdi","
.mdf",".chd"],"report-output":"./igir_%YYYY-%MM-%DDT%HH:%mm:%ss.csv","dat-threads":2,"writer-threads":4,"write-retry":2,"$0":"igir"}
[19:06:01.332] TRACE: DATScanner: scanning DAT files
[19:06:01.342] TRACE: DATScanner: found 1 DAT file
[19:06:01.342] TRACE: DATScanner: enumerating DAT archives
[19:06:01.509] TRACE: DATScanner: parsing 1 DAT file
[19:06:02.171] TRACE: DATScanner: /dev/shm/mame.xml: attempting to parse 278.0MiB of XML
[19:06:20.510] TRACE: DATScanner: /dev/shm/mame.xml: parsed XML, deserializing to DAT
[19:06:24.168] TRACE: DATScanner: /dev/shm/mame.xml: 327.2GiB of 48,574 games, 21,371 parents parsed
[19:06:24.170] TRACE: DATScanner: done scanning DAT files
✓ Scanning for DATs » 1 DAT found
✓ Scanning for DATs » 1 DAT found
[19:06:24.303] TRACE: ROMScanner: scanning ROM files
[19:06:24.396] TRACE: ROMScanner: found 16,229 ROM files
[19:07:27.598] WARN: ROMScanner: /home/nipsy/mame/roms/atronic/atronic.chd: found no files in path
followed by what looks to be all the same offenders as before. I'm not sure I'll let it finish at its current rate!
Definitely no need to let it finish. And I'm about to release the speedup suggestion from https://github.com/emmercm/igir/issues/1776.
Can you post some chdman info --input <file.chd> of CHDs that scan successfully? The ones that are giving you trouble are all CD-ROMs, so now I'm wondering if there's something wrong with the .bin/.cue parsing.
@emmercm Check out https://gist.github.com/nipsy/02f82bc5d2c23386b1c9d8b33f86848e. Let me know once you've grabbed that and I'm going to remove it.
Replying here in order to prevent losing the thread of this conversation. You had mentioned something possibly being wrong with the format of the extracted cue file. My somewhat truncated response follows.
% chdman extractcd -o /dev/shm/sheet.cue -i atronic/atronic.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.278 (unknown)
Input CHD: atronic/atronic.chd
Output TOC: /dev/shm/sheet.cue
Output Data: /dev/shm/sheet.bin
Extraction complete
% cat sheet.cue
FILE "sheet.bin" BINARY
TRACK 01 MODE1/2048
INDEX 01 00:00:00
% file sheet.bin
sheet.bin: UDF filesystem data (version 1.5) 'ATRONIC'
If there is something else to check here, let me know. All of that looks to be fine. I didn't go so far as trying to mount the UDF file on a loopback device. I can if you think I should.
Beyond that, I guess my real question with all of this is, why are we doing this process exactly? I thought the usual verification process for this is simply to compare the hash for the disk from the XML:
<machine name="atronic" sourcefile="misc/atronic.cpp">
<disk name="atronic" sha1="3335e9f8f67f1b176e043f078456d2b13178b7ef" region="cdrom" index="0" writable="no" optional="yes"/>
against the hash returned by chdman:
% chdman info -i atronic/atronic.chd | grep ^SHA1
SHA1: 3335e9f8f67f1b176e043f078456d2b13178b7ef
What purpose does extracting anything further serve in this situation?
I can understand it if it's some form of deeper sanity check potentially. But in that case, it seems like it should be optional anyway as we're simply trusting chdman otherwise.
Let me know what you think. Thanks!