headless-chrome-crawler icon indicating copy to clipboard operation
headless-chrome-crawler copied to clipboard

Published 20 hours ago • verified publisher icon yujiosaka

Crawling site with maxDepth > 2 causes hang

Open TheTFo opened this issue 4 years ago • 3 comments

I'm crawling a small site with maxDepth === 2, and things crawl fine. As soon as up it to 3 or more, the the crawler hangs. I don't see onError or onSuccess called, or any errors. Looking through Fiddler, I do not see any request firing aside from the first batch. How should I troubleshoot this?

What is the current behavior? Crawler seems to hang with no error when maxDepth > 2. It's a rather small site.

If the current behavior is a bug, please provide the steps to reproduce Setting up to queue a particular site with maxDepth > 2 causes things to hang

What is the expected behavior? With any depth, the result should be the same.

Please tell us about your environment:

  • Version: 1.8.0
  • Platform / OS version: MacOS 11
  • Node.js version: 12.8.4

TheTFo avatar Dec 18 '20 02:12 TheTFo

i had the same problem, it was CPU usage for me on a server with 2 cores. with 8 cores i had no problems. maybe that helps.

michaelpapesch avatar Nov 23 '21 17:11 michaelpapesch

You can try adding more memory to node --max_old_space_size=4000

iamprageeth avatar Jun 22 '22 10:06 iamprageeth

I had a similar issue, and fixed it with "maxConcurrency": 1.

goldingdamien avatar Mar 10 '23 04:03 goldingdamien