colly
colly copied to clipboard
gocolly
Elegant Scraper and Crawler Framework for Golang
When I try to use colly/debug this error is occured: ``` Cannot use '&debug.LogDebugger{}' (type *LogDebugger) as the type debug.Debugger Type does not implement 'debug.Debugger' need the method: Event(e *Event)...
[cariddi](https://github.com/edoardottt/cariddi) is using colly. Thank you for all the amazing things you're building. Really thank you.
Before this PR: after cancelling a context for a collector all requests failed with "context canceled" but the delay set by the limit rules still applies. The intention here is...
Robots.txt is ignored by default, as far as I can see: https://github.com/gocolly/colly/blob/master/colly.go#L348-L354 This is disrespectful to site owners. There is no good reason this should be ignored by default: -...
We're trying to build some link checking functionality using colly. But colly on encountering links like "https://www.envoyproxy.io" gives a "301 moved permanently" error even though it is perfectly viewable in...
If colly is crawling a lot of different websites, the robots.txt map can get to be quite large. How would you feel about a PR to use an LRU cache...
Hi there, thank you for your amazing job, it's really a great framework! I'm scratching my head several days but can't understand what's wrong ... I'm using HTTP Forward Proxy...
colly is a very good Scraping Framework。 At the same time, we are also using it on a large scale, but due to the uncertainty of the web page, I...
