EmulationStation
EmulationStation copied to clipboard
RetroPie
Idea to incorporate sselph/scraper in EmulationStation
I had an idea that could help reduce the extra code and maintenance for something that used my code in the existing ES UI. I could write a simple web server that receives the rom path and returns the data that is processed by ES. Similar to https://github.com/Aloshi/EmulationStation/commit/2bd9062c2dcde0c7d2105d48c0c6249ca168cf30 but the hashing takes place in my Go code so I can maintain the bulk of the logic in my repo. It is more code and doesn't work as well for multi file formats(bin/cue). It is also slower than my plain script since the ES code is has to update UI elements, but it might be a better UX for people that don't read the wiki.
I opened this just for discussion. If it is something you wanted to pursue, we could discuss more in detail about the approach. I don't think it would take much on my side to spin something up since Go has standard libraries for creating a web server.
I personally think its a great idea seeing as your scraper is lightyears ahead of the built in scraper especially with all the recent no-intro updates.
I'm a terrible coder but I am a good tester ;)
I don't read c++ well so that would probably be the most time consuming part. Luckily retropie only runs on linux so I don't have to worry about figuring out if the build works in windows. The only concern I have is if the scraper returns no results, I think the current logic is to prompt the user unless an option is set to not do this. We'd probably want to set this option always for my scraper or have a trigger from the scraper that a match isn't possible and skip regardless of the setting. I'll try to review the ES code again.
I was able to write a proof of concept for this. It is a small web server that runs and has an api exposed /getgame?path=
Only issue is even if I return 1 result ES pauses like there are more than one and is asking me to choose like there is a conflict. So you have to turn off the user decides on conflict. I'll continue to play around with it.
Actually, I just noticed there is already a provision for automatically accepting a result if there is a CRC match. I'd just need to signal the match. I also have code already written to do the types of searches that ES does so could potentially try and match the hash, and if it fails fallback to the standard behavior.
Okay I have this in a working state. I implemented 2 scrapers one that is similar to what I do today and just scrapes everything without prompting and one with a manual drop out when things aren't automatically scraped and that mimics the current scraping logic.
What is the best way to proceed with some testing? I can check in the ES code to my fork, send a PR, send a patch. etc. I can check in the code for the scraper web service or post a binary.
For me if you submit a pull request I'm happy to give it a test, I'm not sure what @joolswills prefers or what his thoughts are on the matter
Yeah I guess @joolswills 's opinion is important. My main concern is that it does complicate the system by requiring a second binary to be running. So installing, updating, and making sure it is running. But it does have the benefit of offloading most of the scraping logic.
Tomorrow evening I'll send a PR to this repo and commit my service with some instructions to test it.
Created https://github.com/RetroPie/EmulationStation/pull/51
I uploaded versions of the scraper utility here: https://storage.googleapis.com/stevenselph.appspot.com/web_linux_386.zip https://storage.googleapis.com/stevenselph.appspot.com/web_linux_amd64.zip https://storage.googleapis.com/stevenselph.appspot.com/web_rpi.zip https://storage.googleapis.com/stevenselph.appspot.com/web_rpi2.zip https://storage.googleapis.com/stevenselph.appspot.com/web_windows_386.zip https://storage.googleapis.com/stevenselph.appspot.com/web_windows_amd64.zip https://storage.googleapis.com/stevenselph.appspot.com/web_mac_386.zip https://storage.googleapis.com/stevenselph.appspot.com/web_mac_amd64.zip
So you'd make it work by running the web application in the zip and the updated ES. Then choose sselph or sselph w/fallback. The plain sselph is the one that just runs and the w/fallback will run until it hits something it can't scrape then show the search results with the option to update the search like the current scraper does it.
So I have been revisiting this and have been able to compile my code as a C library that I imported it into ES and was able to use my code directly from ES. The UI could use some improvements, but it works. Now it comes down to what is the easiest way to update the build to allow for it.
From my side I will create a new project or folder that is the library with a make file. It can compile and install the library on linux. This will require Go1.5+ which isn't the default in wheezy. It is super easy to install but it is another step. So I could also provide pre-compiled libraries via github, etc. I would also update ES's CMake to find the installed library.
There are a couple issues: Windows - I know RetroPie isn't supported on windows but I know people use it for development. I have no way of testing builds on it. I will see if my ancient XP CD still works and experiment. I know for a fact I can't build a .dll but I might be able to build a static library. Another option is I could create a dummy library that has the same API but does nothing that can be included in ES and compiles when my library isn't there. Another option is to figure out how to dynamically load the library(dlopen/LoadLibrary) so even if it is missing ES works but has less options. This would be ideal but getting to the limits of my C++ knowledge.
If I provide pre-compiled libraries, I need to know the architectures and be able to compile for them. At the moment the only things I have access to are linux/arm(retropie image) and freebsd/amd64 so I could definitely do the standard retropie on a raspberry pi libraries but others may have to compile from source or we do the dynamic thing mentioned above.
by default in wheezy did you mean jessie? retropie builds on raspbian Jessie, we actually no longer even support wheezy.
Ideally I would think we would want to be able to build it as part of the emulationstation module so you wouldnt have to create precompiled libraries and there are functions for specifying platforms (pi vs x86 etc) Though I presume there may be some systems that use emulationstation that aren't using retropie or if people just choose to use it standalone that might benefit from precompiled libraries- though I don't personally know of anybody.
As far as windows goes, I personally don't really use it all the often though I know there are some in the retropie community that have been doing tests with emulationstation on windows.
As always I'm happy to test implementations
Oops yeah it isn't the default in either :) wheezy used 1.2.1 and jessie uses 1.3.3 I think. The install is: wget https://storage.googleapis.com/golang/go1.7.4.linux-armv6l.tar.gz sudo tar -C /usr/local -xzf go1.7.4.linux-armv6l.tar.gz if path doesn't contain /usr/local/go/bin add it
So it is fairly straightforward. I could probably write some bash that could detect and remove the apt-get version then install the modern version. Once that is in place the build is trivial. So it could probably be hooked in to the dependency area of the emulationstation.sh file.
yeah the go version can be easily sorted as a dependency so that should be the least of our troubles.
Sounds good. I just finished a large change to my code to better handle ScreenScraper. Once I get the library code working with that, I'll check it in. Then will follow up with a fork of EmulationStation and RetroPie-Setup that can handle building the new code.
I'll be happy to test when it's ready. Primary thing is to get it more functional than the current es scraper, then perhaps it may be worth looking at possible implementations of some of the advanced settings your scraper provides
My idea, if you saw my code in the post on the forum is to add 3 auto scrapers that function similar to my script with the defaults. I'll also, in the same update or possibly a followup, update the current builtin to hash then, if that fails, fallback to a name search like it is currently doing. The options on naming, image type, etc will all be left out. If they are advanced enough to know they want more options they will probably figure out using the script. But it would be nice to continue on the path of improving the native UI if possible.
Okay I think I have something workable. It isn't great since I'm sure there are better ways to handle the installs of binaries.
https://github.com/sselph/RetroPie-Setup
If you pull that then look under the exp you will see a few packages but one is emulationstation-sselph and that pulls my new version of EmulationStation with a dependency on libscraper. I ran it successfully but had issues in that the resulting binary was printing debug log info that isn't checked in. So I'm not sure if my environment is messed up.
Heres the build log with errors for reference, looking at it a bit more:
http://pastebin.com/izX9YiqQ
Thanks. Looks like Go isn't being installed correctly. I'll see if I can figure this out with a fresh install. It is supposed to remove the previously installed Go but I'm also referencing the installed Go with the full path so could potentially ignore the apt-get install versionl.
Oh this just might me not doing the install correctly. You see the Installing 'golang' : never appears. If you were to go to that menu entry and install it then it would probably work. I'll update the script.
I am trying to get a log of the text that flashes when I run sudo ./retropie_setup which usually indicates something its not happy with with a module. but yeah it seems to be a hangup with the go install at some point ( the whole setup script modules make things simpler but at the same time sure can complicate things)
Here it is:
/home/pi/RetroPie-Setup/scriptmodules/supplementary/golang.sh: line 32: syntax error in conditional expression
/home/pi/RetroPie-Setup/scriptmodules/supplementary/golang.sh: line 32: syntax error near `"go1.7.5"'
/home/pi/RetroPie-Setup/scriptmodules/supplementary/golang.sh: line 32: ` if [[ "${version}" >= "go1.7.5" ]]; then'
good catch on that error I missed it. Bash is not my forte. I converted >= to > and I don't see the error.
Cool. Get some popcorn cause it takes 30m or so. I used -v when possible to give you something to look at :)
eh. spoke too soon. new build log errors:
http://pastebin.com/sdBgHLXR
still seems to be hanging up on the go install
well shit was about to say that:
Found VLC: /usr/include/vlc
-- Found scraper: /usr/lib/libscraper.so
-- Found ALSA: /usr/lib/arm-linux-gnueabihf/libasound.so (found version "1.0.28")
-- Configuring done
is the important part.
Seems
mv: cannot stat ‘/home/pi/RetroPie-Setup/tmp/build/libscraper/libscraper.h’: No such file or directory
mv: cannot stat ‘/home/pi/RetroPie-Setup/tmp/build/libscraper/libscraper.so’: No such file or directory
/home/pi/RetroPie-Setup/tmp/build/emulationstation-sselph /home/pi/RetroPie-Setup
Is telling
from the log:
Downloading go1.7.5.linux-armv6l.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to create the file
Warning: /home/pi/RetroPie-Setup/tmp/build/golang/golang.tar.gz: No such file
either didn't download properly or something else? perhaps I'm looking at it wrong. still looking at it