discard2 icon indicating copy to clipboard operation
discard2 copied to clipboard

Published 20 hours ago • verified publisher icon Sanqui

Broken in Firefox

Open TheTechRobo opened this issue 2 years ago • 2 comments

Chrome's RAM usage steadily grows when using it with this. I wanted to see if Firefox did any better - Puppeteer does indeed support it. But it's broken. It can't get past the login page - it never ends up filling the form. I'm not sure why.

TheTechRobo avatar Jun 19 '22 20:06 TheTechRobo

Forgot to mention - Firefox doesn't support changing the proxy from the CLI so the profile would have to be modified.

TheTechRobo avatar Jun 19 '22 21:06 TheTechRobo

For what it's worth, while Chrome's RAM usage rose steadily until the page crashed, Firefox (with DHT for autoscroll and mitmproxy for saving) used a ton of RAM, but it stopped rising after a while and after about 10 hours of scraping the page still hadn't crashed.

Plus, there's about:memory -> Minimise Memory Usage which, if run occasionally, can save 100MB or so.

TheTechRobo avatar Jun 20 '22 12:06 TheTechRobo