colly
colly copied to clipboard
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Hello, I use go-colly to crawl through website while with pagination. I set async to true and parallelism is set to 5. I also set timeout to 1 minute. Though I got client Timeout exceeded while awaiting header error.
I have the same problem. everything is the same as my .Net code by it still get an error
I solved that by using this line;
c := colly.NewCollector( colly.MaxDepth(1), colly.DetectCharset(), colly.Async(true), colly.AllowURLRevisit(), ) c.SetRequestTimeout(120 * time.Second)