colly icon indicating copy to clipboard operation
colly copied to clipboard

Published 20 hours ago • verified publisher icon gocolly

Using async wisely

Open VerusK opened this issue 7 months ago • 0 comments

This PR addresses a critical issue encountered when scraping large websites with over 1 million pages. Previously, goroutines were being spawned without any limit, leading to significant memory bloat. This update introduces a more efficient management system for goroutines, effectively resolving the memory issues.

Regarding this issue https://github.com/gocolly/colly/issues/381

VerusK avatar Dec 28 '23 07:12 VerusK