xbapps
panic due to unhanded condition in scrape.VirtualRealPornSite
Problem
Program crashed when scraping.
Analysis
I think this is due to a simple check on the body return values per this stack overflow post
From my understanding of the log output below, it is occurring here I think if you check the jsonResult var
Potential fix
perhaps something like this
if jsonResult != nil {
// continue parsing..
} else {
// elegantly exit function
}
Raw Logs Below
time="2022-10-28T02:19:56Z" level=info msg="visiting https://vrphub.com/oil-worship-ass/"
panic: interface conversion: interface {} is nil, not string
goroutine 1241669 [running]:
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func1.6(0xc00bef9340?, 0xc014942480)
/go/src/pkg/scrape/virtualrealporn.go:78 +0x485
github.com/gocolly/colly.(*HTMLElement).ForEach.func1(0xc0031663d8?, 0xc01e1a9110)
/go/pkg/mod/github.com/gocolly/[email protected]/htmlelement.go:98 +0x83
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc01e1a90e0, 0xc008d10ec0)
/go/pkg/mod/github.com/!puerkito!bio/[email protected]/iteration.go:10 +0x46
github.com/gocolly/colly.(*HTMLElement).ForEach(0xc0149422a0, {0x140cc1f?, 0x13af7dc?}, 0xc008d10fc8)
/go/pkg/mod/github.com/gocolly/[email protected]/htmlelement.go:96 +0x78
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func1(0xc0149422a0)
/go/src/pkg/scrape/virtualrealporn.go:74 +0x42f
github.com/gocolly/colly.(*Collector).handleOnHTML.func1(0xc003166388?, 0xc01e1a8d80)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:963 +0x78
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc01e1a8d50, 0xc008d11280)
/go/pkg/mod/github.com/!puerkito!bio/[email protected]/iteration.go:10 +0x46
github.com/gocolly/colly.(*Collector).handleOnHTML(0xc007f56820, 0xc00bef9340)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:953 +0x24f
github.com/gocolly/colly.(*Collector).fetch(0xc007f56820, {0xa?, 0x3?}, {0x13b0737, 0x3}, 0x1, {0x0?, 0x0}, 0x41607f?, 0xc003a89860, ...)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:623 +0x6ad
github.com/gocolly/colly.(*Collector).scrape(0xc007f56820, {0xc00cdc7090, 0x43}, {0x13b0737, 0x3}, 0x1, {0x0, 0x0}, 0x0, 0x0, ...)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:535 +0x6b1
github.com/gocolly/colly.(*Collector).Visit(0xc00cdc7090?, {0xc00cdc7090?, 0x13af7dc?})
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:412 +0xa6
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func4(0xc01bbde780?)
/go/src/pkg/scrape/virtualrealporn.go:211 +0xe5
github.com/gocolly/colly.(*Collector).handleOnHTML.func1(0xc019fb7630?, 0xc003a89800)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:963 +0x78
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc003a897d0, 0xc008d11758)
/go/pkg/mod/github.com/!puerkito!bio/[email protected]/iteration.go:10 +0x46
github.com/gocolly/colly.(*Collector).handleOnHTML(0xc007f569c0, 0xc0184c5a80)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:953 +0x24f
github.com/gocolly/colly.(*Collector).fetch(0xc007f569c0, {0xa?, 0x3?}, {0x13b0737, 0x3}, 0x1, {0x0?, 0x0}, 0xc01fcf1f00?, 0xc00839c780, ...)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:623 +0x6ad
github.com/gocolly/colly.(*Collector).scrape(0xc007f569c0, {0xc018657fc0, 0x39}, {0x13b0737, 0x3}, 0x1, {0x0, 0x0}, 0x0, 0x0, ...)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:535 +0x6b1
github.com/gocolly/colly.(*Collector).Visit(0xc00595b200?, {0xc018657fc0?, 0x58?})
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:412 +0xa6
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func3(0xc015c92e40?)
/go/src/pkg/scrape/virtualrealporn.go:203 +0x75
github.com/gocolly/colly.(*Collector).handleOnHTML.func1(0xc00069a678?, 0xc00839c690)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:963 +0x78
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc00329b9b0, 0xc008d11bf0)
/go/pkg/mod/github.com/!puerkito!bio/[email protected]/iteration.go:10 +0x46
github.com/gocolly/colly.(*Collector).handleOnHTML(0xc007f569c0, 0xc0008174c0)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:953 +0x24f
github.com/gocolly/colly.(*Collector).fetch(0xc007f569c0, {0xa?, 0x3?}, {0x13b0737, 0x3}, 0x1, {0x0?, 0x0}, 0x29?, 0xc007f55500, ...)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:623 +0x6ad
github.com/gocolly/colly.(*Collector).scrape(0xc007f569c0, {0xc004368690, 0x29}, {0x13b0737, 0x3}, 0x1, {0x0, 0x0}, 0x0, 0x0, ...)
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:535 +0x6b1
github.com/gocolly/colly.(*Collector).Visit(0xc007f569c0?, {0xc004368690?, 0x19?})
/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:412 +0xa6
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite(0x0?, 0x1, {0xc0088ec000, 0x53ce, 0x5c00}, 0xc0086d41e0, {0x13d2ed0, 0xf}, {0x13d2138, 0
xf}, ...)
/go/src/pkg/scrape/virtualrealporn.go:218 +0x48a
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPorn(0x0?, 0x0?, {0xc0088ec000?, 0x0?, 0x0?}, 0x0?)
/go/src/pkg/scrape/virtualrealporn.go:231 +0x57
created by github.com/xbapps/xbvr/pkg/tasks.runScrapers
/go/src/pkg/tasks/content.go:109 +0x49b
If your log is correct, I don't understand how you are scraping a vrphub.com scene with the virtualrealporn scraper, the virtualrealporn scraper does not scrape that site. I'm not surprised it didn't work as vrphub pages will be completely different and if the scraper didn't crash there it probably would somewhere else.
Do you have any mods or redirects. Are you sure the log message with the vrphub.com scene is the one that caused the crash. Maybe try scraping just the one single site (option on the 3 dots) rather than all of them to confirm the site with a problem. I'm not aware of anyone else having issues with this.
@wheresmytissue cleaning up "dead" issues. VirtualRealPorn scraper has been updated (not for this issue) if you still have the problem, reopen with more information.
whoops.. jumped the gun on this one.. (scraper was updated day before issue logged).
@wheresmytissue Please give more information (still an issue? xbvr version? answer @toshski, etc..) or I will close again in a day or two.