web-scraping-parallel-processing
web-scraping-parallel-processing copied to clipboard
Memory drain
Hi Caleb
Was wondering if you tried using this for many thousands of websites. I adapted what you wrote, but the geckodrvier and marionette processes just keep increasing in memory until my pc stops responding. It seems that memory and swap never get released, but I'm not entirely sure. Limiting the number of workers seems to offer no relief either.
Ideas ?