Adjust code to new ScreenScraper API

[Torben2000]

With these changes, scraping via ScreenScraper works again for me.

Both the URL and the whole response structure changed, so there was quite a lot to do.

I am not a Go expert, so any comments concerning the code are well appreciated.

This pull request should fix #263, #251 and #262

[Cereal-Killa]

Would you happen to have windows binaries of your build to share?

[Torben2000]

https://github.com/Torben2000/scraper/releases/tag/v1.5.0-pre I have no idea how to build a "real" version (this one seems to be a debug one), so I marked it as pre-release.

[Cereal-Killa]

https://github.com/Torben2000/scraper/releases/tag/v1.5.0-pre I have no idea how to build a "real" version (this one seems to be a debug one), so I marked it as pre-release.

Thank you, I gave it a shot and the output was:

Get "https://storage.googleapis.com/stevenselph.appspot.com/openvgdb2.zip": dial tcp: lookup storage.googleapis.com: getaddrinfow: A non-recoverable error occurred during a database lookup.

What could that mean since that url is actually valid?

[Torben2000]

I don't use GDB at all and I did not change anything in that part (at least not on purpose). My guess is that there is another bug concerning GDB.

[J-Swift]

Looks like a transient HTTP issue to me. Did you try it again since?

[Cereal-Killa]

Looks like a transient HTTP issue to me. Did you try it again since?

There you go, it was a transient issue and the scraping simply worked now that I tried again. Thanks!!

[MrBullfinsh]

Does anybody have an idea how to build this version? This is the first time I get in touch with go and I managed to build the official version via go get github.com/sselph/scraper etc. But when I try to build it from cloned source I get the same error for the official version and for this version. What I did:

git clone https://github.com/Torben2000/scraper
cd scraper
git checkout feature/newApiSS
go mod init scraper
go mod tidy
go get ./...
go mod vendor
GOARM=7 GOARCH=arm GOOS=linux go build

What I get:

# scraper
./scraper.go:654:28: too many arguments in call to gdb.IsUp
        have (context.Context, string)
        want (context.Context)
./scraper.go:659:76: unknown field 'APIKey' in struct literal of type ds.GDB
./scraper.go:660:64: unknown field 'APIKey' in struct literal of type ds.ScummVM
./scraper.go:661:63: unknown field 'APIKey' in struct literal of type ds.Daphne
./scraper.go:662:63: unknown field 'APIKey' in struct literal of type ds.NeoGeo
./scraper.go:715:61: unknown field 'APIKey' in struct literal of type ds.NeoGeo

I'm using a Raspberry Pi 3 Mod. B with RetroPie v4.8 and go version go1.18 linux/arm . Thanks very much in advance for any help!

[J-Swift]

@MrBullfinsh I'm not very experience with Go but I think the go modules might be messing you up. Heres a command I had written down long ago when I built it (I think its for Synology NAS though, so you will have to tweak the architecture):

GOPATH=$(pwd) GOOS=linux GOARCH=arm64 go build  -o scraper_arm64 src/github.com/sselph/scraper/scraper.go

[Cereal-Killa]

Does anybody have an idea how to build this version? This is the first time I get in touch with go and I managed to build the official version via go get github.com/sselph/scraper etc. But when I try to build it from cloned source I get the same error for the official version and for this version. What I did:

git clone https://github.com/Torben2000/scraper
cd scraper
git checkout feature/newApiSS
go mod init scraper
go mod tidy
go get ./...
go mod vendor
GOARM=7 GOARCH=arm GOOS=linux go build

What I get:

# scraper
./scraper.go:654:28: too many arguments in call to gdb.IsUp
        have (context.Context, string)
        want (context.Context)
./scraper.go:659:76: unknown field 'APIKey' in struct literal of type ds.GDB
./scraper.go:660:64: unknown field 'APIKey' in struct literal of type ds.ScummVM
./scraper.go:661:63: unknown field 'APIKey' in struct literal of type ds.Daphne
./scraper.go:662:63: unknown field 'APIKey' in struct literal of type ds.NeoGeo
./scraper.go:715:61: unknown field 'APIKey' in struct literal of type ds.NeoGeo

I'm using a Raspberry Pi 3 Mod. B with RetroPie v4.8 and go version go1.18 linux/arm . Thanks very much in advance for any help!

For Windows: https://krakenfiles.com/view/Mi6xr7COfq/file.html

[tloudon]

@MrBullfinsh

try:

go mod init github.com/sselph/scraper
go get github.com/J-Swift/thegamesdb-swagger-client-go@f9a22a4e0cd4
go mod tidy
go get ./...
go mod vendor
GOARM=7 GOARCH=arm GOOS=linux go build

if your go.mod just has module scraper, it won't use the local files and will instead download and use the repo at version 1.4.6 (the imports in scraper.go all reference github.com/sselph/scraper not just scraper).

@J-Swift also updated his module, so you need to use the version linked or update the local code to use the new function signatures, etc (see https://github.com/J-Swift/thegamesdb-swagger-client-go/commit/06967c20ca1d18e7fbba5929d1b4bf930052956d#commitcomment-75241057)

[J-Swift]

Sorry about that breakage, I didn't realize it was going to affect downstream immediately. As mentioned in that link, I am actually doing a sort of from scratch rewrite of this project. Its almost ready for beta status, but missing some big things (non gamesDB apis, arcade, bin/cue, lots of CLI flags which I probably don't plan to port).

[tloudon]

@J-Swift no need to apologize--perfectly reasonable to update your own repo, especially after 2 years!

I'm not a gopher, but I think the way this is typically handled is by checking both go.mod and go.sum into the repo--then dependencies can be pinned to a specific version.

If your project is still a ways out, maybe a simple PR to @Torben2000 w/ an updated go.mod and go.sum and vendor dir--since it's already checked into the repo--would do the trick, yeah? I'm happy to spend five minutes creating the PR if it will be merged.

TL;DR You shouldn't feel like you broke anything or are obligated to finish a project :). Thanks for the fixes on the gamesdb API.

[0de11]

Does anybody have an idea how to build this version? This is the first time I get in touch with go and I managed to build the official version via go get github.com/sselph/scraper etc. But when I try to build it from cloned source I get the same error for the official version and for this version. What I did:

git clone https://github.com/Torben2000/scraper
cd scraper
git checkout feature/newApiSS
go mod init scraper
go mod tidy
go get ./...
go mod vendor
GOARM=7 GOARCH=arm GOOS=linux go build

What I get:

# scraper
./scraper.go:654:28: too many arguments in call to gdb.IsUp
        have (context.Context, string)
        want (context.Context)
./scraper.go:659:76: unknown field 'APIKey' in struct literal of type ds.GDB
./scraper.go:660:64: unknown field 'APIKey' in struct literal of type ds.ScummVM
./scraper.go:661:63: unknown field 'APIKey' in struct literal of type ds.Daphne
./scraper.go:662:63: unknown field 'APIKey' in struct literal of type ds.NeoGeo
./scraper.go:715:61: unknown field 'APIKey' in struct literal of type ds.NeoGeo

I'm using a Raspberry Pi 3 Mod. B with RetroPie v4.8 and go version go1.18 linux/arm . Thanks very much in advance for any help!

For Windows: https://krakenfiles.com/view/Mi6xr7COfq/file.html

Thank you so much for this! All I did was replace the scraper.exe in my $USER$/go/bin directory and everything magically worked. I appreciate everyone involved for resurrecting this tool!

[GiovanH]

For Windows: https://krakenfiles.com/view/Mi6xr7COfq/file.html

What did you actually do to build this?

[Cereal-Killa]

For Windows: https://krakenfiles.com/view/Mi6xr7COfq/file.html

What did you actually do to build this?

I just had to setup a go development environment, the instructions to build are in the repo