web-scraping-parallel-processing icon indicating copy to clipboard operation
web-scraping-parallel-processing copied to clipboard

Memory drain

Open dejmail opened this issue 3 years ago • 0 comments

Hi Caleb

Was wondering if you tried using this for many thousands of websites. I adapted what you wrote, but the geckodrvier and marionette processes just keep increasing in memory until my pc stops responding. It seems that memory and swap never get released, but I'm not entirely sure. Limiting the number of workers seems to offer no relief either.

Ideas ?

dejmail avatar Aug 28 '22 19:08 dejmail