colly icon indicating copy to clipboard operation
colly copied to clipboard

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Open ru90 opened this issue 2 years ago • 2 comments

Hello, I use go-colly to crawl through website while with pagination. I set async to true and parallelism is set to 5. I also set timeout to 1 minute. Though I got client Timeout exceeded while awaiting header error.

ru90 avatar Aug 03 '21 10:08 ru90

I have the same problem. everything is the same as my .Net code by it still get an error

sbdevman avatar Oct 14 '21 14:10 sbdevman

I solved that by using this line; c := colly.NewCollector( colly.MaxDepth(1), colly.DetectCharset(), colly.Async(true), colly.AllowURLRevisit(), ) c.SetRequestTimeout(120 * time.Second)

sbdevman avatar Oct 14 '21 14:10 sbdevman