scraper
scraper copied to clipboard
sselph
Support for Psx, PSP
Hello, sorry for my english, is it possible to add support for psx, psp game ? your tool is great ! thank for your work :D
Hi @sselph , I'd also like to see PSX supported, seeing as the Raspberry Pi can run a lot of the games pretty reliably. If you need assistance in matching games to TGDB IDs (with help from your Reporting tool) then I'm happy to help.
Hi @sselph Great tool; I'd also really appreciate PSX support - the Pi 2 is virtually flawless on everything I've used, so I am building quite a library of PlayStation games. Thanks
I have looked at PSX support. The issue was that the collection of hashes I found was for .iso files but seemed like most people had bin/cue files. It seemed like it might be possible to convert to an ISO from BIN/CUE but couldn't find a ton of information on the file formats themselves so don't know if I could consistently convert to the same ISO that was used to create the hashes.
@robertybob if you would like to create the mapping for .bin files on PSX to TGDB IDs I definitely won't mind. If those are create it is just a matter of me adding them to the DB and uploading them.
@sselph Will do! I'll upload one .csv file here. I can't tell you, however, whether a game is an USA or Europe etc release, I presume your tool will figure that one out :) Also, if it's a multi-dsc game, will an entry be created for each disc? There is a tool to convery multiple .bin files into one but I can't remember what it's called.
@sselph I've only got 50 games in my collection (out of the 1300 that TGDB lists) and unfortunately I don't have the time or storage space to told that many games at any one time. Would you like me to ask the community (e.g. people like @retromancer) to use your reporting tool and send the CSV of hashes to me so I can collate them?
My hashes: https://drive.google.com/file/d/0ByWwZdQX1FQmM3hCYWdnWkZUQTA/view?usp=sharing
Let me start by adding the iso hashes I can find and adding recognition of the iso file extension. Then I'll do some more investigating on the different file formats. Maybe I can find some time to do that this evening. If I can't figure out the file conversion I can start seeing if people can donate hashes for the other file formats. If there are 1300 games it will probably take me a couple evenings to match all the games.
What I could do at some point this week is look into software such as MagicISO which claims to convert .bin to .iso - I convert a couple of games to .iso and see whether the hashes are the same.. if they are, then that should make things a lot easier for all!
Actually I got that backwards I have the hashes for bin/cue. I'll go ahead and get those added. Then see if it is possible to convert a iso/img to a bin so I can hash it. From what I understand, the iso is similar to a bin but with less information so that transformation might not be possible, but we will at least have partial support.
Sounds good. I've found a few more .bin files but when I run the reporting tool it doesn't appear in the missing.csv and obviously when I run the scraper it doesn't appear in gamelist.xml so does that mean it's a bad dump?
Also, is your list of hashes from here? http://redump.org/discs/system/psx/region/Am/ http://redump.org/discs/system/psx/region/Eu/ I note that it has the sizes, MD5 and SHA-1 hashes, so for games I don't own I could manually add these (with their TGDB ID) to a spread sheet as that is the data presumably pulled by your reporting tool?
Also, having so far only got PSX games with single discs, I can't get my head around how to scrape games with multiple .bin files [ see here ] without creating multiple entries of the same game in the gamelist.xml . Have you any thoughts ? How do we tackle games like Metal Gear Solid which has multiple discs? http://blog.petrockblock.com/forums/topic/best-way-for-psx-multidisc-games/
Yeah that is what I ran in to last night and have an idea. I'll create a special flag for bin/cue scraping that will first scrape the cue file and if it matches a hash add it. But the cue is just a text file with all the names of the bin files in them so if the file name doesn't match the expected name it won't match the hash so I'll then hash the bin files to see if there is a match but only add an entry for the cue files. Users will have to remove .bin from the extensions allowed or turn on the option that only pulls in things from the gamelist. As for multi disc games I'm just going to use the naming in the dat file so Rom (Disc 1), Rom (Disc 2) then looks like I might need to figure out how to look in side pbp files to find rom data.
That sounds promising. Is there anything I can do to help in the mean time? Would you still like more TDGB's matched to .bin hashes?
Let me get the original set created and we can see if there is anything missing. It will take me a couple days since there are 1300 games.
https://docs.google.com/spreadsheets/d/1nd4Kp7c4fQMhS00xBmiFI4VNIYAbUp-u3O-N52TvGKQ/edit?usp=sharing
This is my work in progress. Feel free to request permission to edit. I have some scripts and Vlookups to match most the games by name to the name in tgdb. I now go through and look at all tgdb games to make sure they all match a game in the redump DB.
I've started editing the spread sheet- only problem is I entered new games at the bottom of the TGDB tab and matched the names on the Redump tab - however they weren't turning green. Wasn't sure if I was doing something wrong or if the conditional formatting hadn't copied down correctly (?)
I've completed matching all TGDB games to redump games that I can find. As I was going I looked for very similar non-english titles and matched them as well. If you have more games you want to add let me know. After that I'll convert it to csv and add it to the others.
I've pushed out all the changes that should allow the bin/cue PSX stuff to work. Let me know if it does.
Brilliant! If you haven't already done so, I think you should great a new post on the RetroPie forums to inform everyone of the recent huge additions (DC & PSX) ! :)
No problem. I'm keeping the issue open to continue to track psp but I haven't looked at that yet.
Hi @sselph so what rom set matches the hashes you have in the scraper? The Redump set?
Thanks for the reply mate! What other systems are being added? MSX would be great! On 14 Jul 2015 8:11 am, "Steven Selph" [email protected] wrote:
Yes it is the redump set in bin/cue format.
— Reply to this email directly or view it on GitHub https://github.com/sselph/scraper/issues/30#issuecomment-121077280.
@sselph I just wanted to update you on PSP and PSX. I am finishing my Redump sets on these and everything else over the summer. Once done I will look at getting hashes for all of these for USA. The set is huge though so it will take me awhile.
PSX scraping of cue/bin files works great. Any progress on PSP games and PSX .iso files?
I haven't done much with PSP or iso's.
Redump has bin/cue hashes for PSX so I can just write code to hash the bins and compare it to their database which is easy, but I don't have a complete set of hashes for the isos. For systems like Genesis, N64, etc that have multiple formats, I'm able to detect the format and convert the file to the format that was originally hashed to get a match against the database.
When I looked last, I couldn't find a good source of information to know if I could consistently convert the information in an iso to what would be in the bin file. I imagine it could be possible since an iso is a container of all the tracks on a CD but just couldn't find good information describing it at the bit level. I thought about taking a few bins of known size and patterns and passing them to a bin>iso tool then opening the files in a hex editor to try and figure it out but never got around to it.