colly
colly copied to clipboard
gocolly
Elegant Scraper and Crawler Framework for Golang
I was writting some code which need the response's **"Proto"** field,but I found that this field seems be deprecated by the colly.Response,Is there any way to get the proto field...
Hello Sir , I'm a Chinese native-speaking developer, and I love Colly very much. I'd like to contribute a Chinese version of Colly documentation to http://go-colly.org/. What can I do?...
When I colly.Post(url,map[string]string{"pwd":"admin","user":"admin","_token":""}) the _pwd in map[string]string will be lost , what is the situation, in the case of single thread and multi-thread, I suspect this is a bug 
Hi, I want to say thanks to all of the contributors and creators for providing this package, the code is very clean and fast. I use colly in my project...
Visiting https://go-colly.org/ shows a different website (omnom) instead of colly's. If it's not intended, when you'll find the time to fix it you could also enable automatic `https` redirection. This...
Hi, This is a question, not a bug report. [url-frontier](https://github.com/crawler-commons/url-frontier) is an API to define a [crawl frontier](https://en.wikipedia.org/wiki/Crawl_frontier). It uses gRPC and has a service implementation. It is crawler-neutral and...
Thanks for maintaining this package! Simple scheduling mechanism to periodically start a `Collector`'s scraping routine. The entire schedule group can be cancelled via `StopSchedules` or cancelling the `Collector.Context`. When a...
Hi, thanks for implementing this tool. The code is clean, and APIs are easy to understand. I was wondering how can I make `maxDepth` work when using `queue API` (http://go-colly.org/docs/examples/queue/)?...
I was debugging some logic I was adding to an OnRequest (stopping a crawl at a timeout) and as I walked through the code I noticed something that felt a...
The default setting is not suitable for long runs
