colly icon indicating copy to clipboard operation
colly copied to clipboard

checkRobots failed for scheme less url

Open 0xTanvir opened this issue 4 years ago • 0 comments

For this line of code https://github.com/gocolly/colly/blob/2f09941613011bfde62cbe4a695310b42bf42d41/colly.go#L784

If url is www.url.com or url.com it will generate :///robots.txt because url.Parse(u) will move the url.com to u.Path() instead of u.Host() due to missing scheme and which is generating wrong URL for robots.txt. It also doesn't return any error.

0xTanvir avatar Nov 25 '21 11:11 0xTanvir