emmercm
Multi-disc gamelist creation
https://wiki.batocera.org/cd_image_formats#multi-disc_games
Some disc-based emulators won't support zip files, so would be nice to generate these kinds of solutions. The problem you see as a user is you end up with multiple copies of a game in the UI (unclear if it launches each disc, havent tried).
May be out of scope for this project, and there may be other formats or other solutions. Super adjacent though and somehow could be nice as part of the copy-to-sorted story.
An improved addition woudl be to allow folder creation per game:
https://wiki.batocera.org/cd_image_formats#a_subfolder_for_each_multi-disc_game
Feels a little gross to do this for every single emulator though just to support disc-based ones in a single command.
Either way it looks like the .m3u is still needed. Maybe could just be an option to generate m3us when there are multi disc games, and an additional option to generate a folder per rom (which maybe you can pull off today w/ the output tokens, didnt eval).
I'm not positive I understood every part of the comment, so let me take a stab and let me know where it's off.
RE: excluding some files from zipping:
There's an option exactly for this, but I'm realizing it's not documented anywhere. To exclude some file extensions you can do:
igir <copy|move> zip ... --zip-exclude "**/*.{iso,bin,cue,chd}"
RE: disc grouping & .m3u creation:
This seems like a good addition, I'll see what I can do. I'll have to think on the best way to accomplish this in code.
@emmercm awesome - happy to take a stab myself if I get around to it, just wanted to make sure it made sense
It makes total sense. I think I remember doing something like this with Lakka a few years ago, which makes sense since basically every frontend is RetroArch/libretro under the hood.
I wonder if this would be better split into two features:
- The ability to group multi-disc games into one game (either taking a DAT and grouping the games within it, or grouping "release candidates", I haven't decided which is best yet -- but either way it would need a new "module")
- The ability to make m3u files for multi-file games (also probably as a new "module")
Something like:
igir {copy|move} --game-group-discs --game-m3u "*.cue"
Writing down a summary of what I'm thinking as part of this:
- Specify certain consoles to unzip. I cant think of a better way than console matching here, as its emulator specific whether something needs unzipped or not. This might not be needed if we can just auto bundle multi-file roms into their own subfolder as bullet point two. This def would be best if we can simply peek the archive. e.g.
--unzip-only="*.cue" - Add a flag to bundle game multi-disc/file game releases (I think we want multi file in this case? e.g. cue/bin go together). I don't know what the DAT stores but if it tells us that theres N discs then the flag you suggested seems like a great approach.
- Add a flag to generate m3u files for ... any game that has multiple discs? Seems like this could be a new command (e.g.
generate-m3uin case it needs additional flags).
For the full process, something like this is what I'm targeting:
copy extract test clean generate-m3u \
--dir-rom-name \
--dat /mnt/roms/Dats/* \
--input "/mnt/roms/Unsorted/" \
--output "/mnt/roms/Sorted/{batocera}" \
--language-filter EN \
--region-filter USA,WORLD
This isnt quite ideal, but what I'd see above is:
- Every rom is placed in its own dir (a child of the --output folder).
- Multi-disc games have m3u's generated and placed in --output.
- Clean operates on the root --output, meaning any game folders that were created that no longer should exist also get removed.
The only real negative to this approach that I can think of is that we're creating a directory for every game which feels a little wasteful. Not sure if this will actually cause any issues. We're also unzipping roms which could otherwise stay zipped, but given the files aer contained in subdirs that feels ok.
Responding to the bullets:
- I think
--zip-exclude <glob>should already do what you want, unless I misunderstood the intent of--unzip-only. I think the.binfiles would also need to stay un-zipped. - If I understood you right, multi-ROM games are actually already bundled into their own folder on output right now, and there is no option to turn that behavior off (yet). So in the case of Redump DATs, they list different discs of a game as different games, i.e.
<game name="Final Fantasy IX (USA) (Disc 1) (Rev 1)">. I think the change needed is a new flag to group the ROMs of these games together in a new game. I'm inclined to do this after DAT scanning and before DAT filtering. - A command would probably make the most sense, I actually regret making fixdat not a command, something I'm fixing in https://github.com/emmercm/igir/pull/566. I like the one-word commands igir has going now, maybe
playlist? I think it may need an option like--playlist-m3u <glob>as in--playlist-m3u "**/*.cue"to specify what files should be in a playlist.
I just realized that it seems to automatically unzip archives which have multiple files into their own subfolder. Assuming thats correct, then this is actually way easier and just needs the m3u generation.
Edit: And upon re-reading your reply you clearly say that 😆
Actually looks like this may be more painful than I hoped... https://forum.batocera.org/d/5216-multi-disc-final-explanation-please/12
e.g. if you have to both generate the m3u and modify this mostly? generated xml file...
Going to try to confirm this today..
Per https://wiki.batocera.org/cd_image_formats#a_subfolder_for_each_multi-disc_game I don't think the XML needs editing, but I'm not familiar at all.
This part specifically:
Batocera will automatically hide duplicated entries showing in the gamelist when using playlists to define which discs are a part of a single game. However, this will only work when the playlists are referencing just the filenames...
Alright digging into this a bit more and just dropping one note here for reference.
Upon extracting, zip files are placed into their own directory (thats good), but its obviously one directory per zip, which is one directory per disc. We will preferably move and cluster them for any semblence of struc ture.
For example:
My Game (Disc 1)/
My Game (Disc 1)/My Game (Disc 1).iso
My Game (Disc 2)/
My Game (Disc 2)/My Game (Disc 2).iso
The outgoing result would preferably look like:
My Game/
My Game/My Game.m3u
My Game/My Game (Disc 1)/
My Game/My Game (Disc 1)/My Game (Disc 1).iso
# etc
That said, we could also leave them in this structure, and just generate the m3u in the root folder here.
Waiting for sync, but testing out a quick run with a Python script to make sure it behaves as expected (with no other changes, just m3u generation):
import re
import os
import os.path
disc_extensions = ('.iso', '.cue')
name_regexp = re.compile(r'^(.+)\s\(Disc \d+\).*$', re.I)
def iter_dirs(root):
for filename in os.listdir(root):
path = os.path.join(root, filename)
if os.path.isdir(path):
yield path
def iter_files(root):
for filename in os.listdir(root):
path = os.path.join(root, filename)
if os.path.isfile(path):
yield filename
def write_playlist(root, name, file_list):
m3u_filename = os.path.join(root, name + '.m3u')
print(f'Writing m3u for {name}: {len(file_list)} discs')
with open(m3u_filename, 'w') as fp:
for name in sorted(file_list):
fp.write(name)
def main(root):
for console in iter_dirs(root):
discovered = {
# "Game Name": filelist
}
# this needs to be sorted :(
for game in sorted(iter_dirs(console)):
match = name_regexp.match(game)
if match:
game_name = match.group(1)
for filename in iter_files(game):
if filename.endswith(disc_extensions):
discovered.setdefault(game_name, []).append(
os.path.join(os.path.basename(game), filename)
)
for game, file_list in discovered.items():
write_playlist(console, os.path.basename(game), file_list)
if __name__ == '__main__':
import sys
main(root=sys.argv[1])
Actually looks like this may be more painful than I hoped... https://forum.batocera.org/d/5216-multi-disc-final-explanation-please/12
e.g. if you have to both generate the m3u and modify this mostly? generated xml file...
Going to try to confirm this today..
You don’t need to modify the XML file. An approach that I take is to change the extension of my CHD or ISO files to “.cd” so the individual disks don’t get scraped. I can then generate my m3u file using these .cd files.
https://github.com/emmercm/igir/issues/587#issuecomment-1691018272
multi-ROM games are actually already bundled into their own folder on output right now, and there is no option to turn that behavior off (yet)
I'm adding the option in https://github.com/emmercm/igir/pull/692.
is if i get the wright this all sounds good and would love to see this used my use case is for Retoarch when importing rom set i need .M3U's to be in the DAT for multi-disc games so i can create a playlist filter out Regions or language using in app features also so the save game features work correctly because when it creates a game for every cue. saves don't cross over to the next disk when you load it.
below is a Linux bash shell script that will scrape the Redump DAT rom folder structure create a new game.m3u and put it in the root folder they only problem i see is adding it to the existing DAT
the M3U's that are created would look like this
PSX\Final Fantasy VII (USA).m3u
|
|
Final Fantasy VII (USA) (Disc 1)/Final Fantasy VII (USA) (Disc 1).cue
Final Fantasy VII (USA) (Disc 2)/Final Fantasy VII (USA) (Disc 2).cue
Final Fantasy VII (USA) (Disc 3)/Final Fantasy VII (USA) (Disc 3).cue
PSX\Final Fantasy VIII (USA).m3u
|
|
Final Fantasy VIII (USA) (Disc 1)/Final Fantasy VIII (USA) (Disc 1).cue
Final Fantasy VIII (USA) (Disc 2)/Final Fantasy VIII (USA) (Disc 2).cue
Final Fantasy VIII (USA) (Disc 3)/Final Fantasy VIII (USA) (Disc 3).cue
Final Fantasy VIII (USA) (Disc 4)/Final Fantasy VIII (USA) (Disc 4).cue
I do believe we need a Dir2Dat function that does understand multi-disk games and when it finds it group them in the same game rom set with token like Disc # or track # or Disk # when is see's this token it knows to group them in the same folder to look like this that way you can just create your own folder structure and it understands how to hand it special use cases this is assuming you dont have a DAT with M3U's and your creating one like i had to.
A group function would me nice to add to your clean / fix / move / copy commands to add this functionality that would look for games with Disc or Track or Disk. in the file or Dir name and provide processing to sort out your DAT to a more logical one.
# ls -R
.:
'Final Fantasy VII (USA)' 'Final Fantasy VIII (USA)'
'./Final Fantasy VII (USA)':
'Final Fantasy VII (USA) (Disc 2).cue' 'Final Fantasy VII (USA) (Disc 2).bin' 'Final Fantasy VII (USA) (Disc 1).bin'
'Final Fantasy VII (USA) (Disc 3).bin' 'Final Fantasy VII (USA) (Disc 1).cue'
'Final Fantasy VII (USA) (Disc 3).cue' 'Final Fantasy VII (USA).m3u'
'./Final Fantasy VIII (USA)':
'Final Fantasy VIII (USA).m3u' 'Final Fantasy VIII (USA) (Disc 3).bin'
'Final Fantasy VIII (USA) (Disc 4).bin' 'Final Fantasy VIII (USA) (Disc 2).cue'
'Final Fantasy VIII (USA) (Disc 1).cue' 'Final Fantasy VIII (USA) (Disc 2).bin'
'Final Fantasy VIII (USA) (Disc 1).bin' 'Final Fantasy VIII (USA) (Disc 3).cue'
'Final Fantasy VIII (USA) (Disc 4).cue'
#
below is bash shell script used to create M3U's
#!/usr/bin/env bash
set -Eeuo pipefail
mapfile -t PsxGames <<< $(ls -d *\(Disc* | sed 's/ (Disc .).*//' | uniq)
for game in "${PsxGames[@]}"; do
echo "Processing '$game'..."
m3u_file=${game}.m3u
if [[ -f $m3u_file ]]; then rm "$m3u_file"; fi
mapfile -t cues <<< $(eval ls ${game@Q}*/*.cue)
for cue in "${cues[@]}"; do
echo $cue >> "${m3u_file}"
done
done
