osg
osg copied to clipboard
Runtime error while crawling `reactivex.io`.
Hello.
I've just tried to crawl http://reactivex.io/documentation because I found out some documentation pages appear to be hidden. Unfortunately, osg
crashed, outputting the following stacktrace with an error signal of 2
:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5787d0]
goroutine 856 [running]:
net/url.(*URL).String(0x0, 0x85, 0x0)
/usr/lib/go/src/net/url/url.go:742 +0x40
main.GetLinks(0x7f4045d63fd0, 0xc420a66bc0, 0xc420166800, 0x0, 0x40b78f, 0xc420461ca0)
/home/vrakfall/go/src/github.com/patrickmn/osg/main.go:290 +0x2f6
main.(*Crawler).GetLinks(0xc42014e2c0, 0x7f4045d63fd0, 0xc420a66bc0, 0xc420166800, 0xc420166080, 0xc420c7b860, 0x19, 0x0)
/home/vrakfall/go/src/github.com/patrickmn/osg/main.go:171 +0x5a
main.(*Crawler).Crawl(0xc42014e2c0, 0xc420166800, 0xc420166080, 0x0, 0x0)
/home/vrakfall/go/src/github.com/patrickmn/osg/main.go:147 +0x52d
created by main.(*Crawler).Crawl
/home/vrakfall/go/src/github.com/patrickmn/osg/main.go:154 +0x5c2
Osg
found some dead links normally returning a 404 status just before crashing. This does look like a kind of memory overflow to me (I don't know if that can happen in go
, my knowledge of it being limited.).
It also worked fine on another website I tried. The error seems linked to reactivex.io
. Maybe there are too many sublinks? I'm sure it's not infinite tho, I tested it with another script.