osg icon indicating copy to clipboard operation
osg copied to clipboard

Runtime error while crawling `reactivex.io`.

Open Vrakfall opened this issue 6 years ago • 0 comments

Hello.

I've just tried to crawl http://reactivex.io/documentation because I found out some documentation pages appear to be hidden. Unfortunately, osg crashed, outputting the following stacktrace with an error signal of 2:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5787d0]

goroutine 856 [running]:
net/url.(*URL).String(0x0, 0x85, 0x0)
        /usr/lib/go/src/net/url/url.go:742 +0x40
main.GetLinks(0x7f4045d63fd0, 0xc420a66bc0, 0xc420166800, 0x0, 0x40b78f, 0xc420461ca0)
        /home/vrakfall/go/src/github.com/patrickmn/osg/main.go:290 +0x2f6
main.(*Crawler).GetLinks(0xc42014e2c0, 0x7f4045d63fd0, 0xc420a66bc0, 0xc420166800, 0xc420166080, 0xc420c7b860, 0x19, 0x0)
        /home/vrakfall/go/src/github.com/patrickmn/osg/main.go:171 +0x5a
main.(*Crawler).Crawl(0xc42014e2c0, 0xc420166800, 0xc420166080, 0x0, 0x0)
        /home/vrakfall/go/src/github.com/patrickmn/osg/main.go:147 +0x52d
created by main.(*Crawler).Crawl
        /home/vrakfall/go/src/github.com/patrickmn/osg/main.go:154 +0x5c2

Osg found some dead links normally returning a 404 status just before crashing. This does look like a kind of memory overflow to me (I don't know if that can happen in go, my knowledge of it being limited.).

It also worked fine on another website I tried. The error seems linked to reactivex.io. Maybe there are too many sublinks? I'm sure it's not infinite tho, I tested it with another script.

Vrakfall avatar Jun 27 '18 17:06 Vrakfall