colly
colly copied to clipboard
checkRobots failed for scheme less url
For this line of code https://github.com/gocolly/colly/blob/2f09941613011bfde62cbe4a695310b42bf42d41/colly.go#L784
If url is www.url.com or url.com it will generate :///robots.txt because url.Parse(u) will move the url.com to u.Path() instead of u.Host() due to missing scheme and which is generating wrong URL for robots.txt. It also doesn't return any error.