Improvement: RAM Allocation

[3dycosmo]

I using webtopeub on chrome (latest version) for lnmtl.com for 3 or 4 novels at the same time and then it become like snail (and come out warning for wait or kill tab option), albeit chrome's other tabs work normal because my RAM still 40% usage. *Perhaps chrome maintains the extension for RAM usage as well as RAM each tabs used, could you bypass it and used all RAM directly

[3dycosmo]

have been implemented on #304 and webtoepub also adjusting to delay packaging an epub if many webtoepub runs so not to make packaging epubs at the same time for multiple epubs

[dteviot]

With regards to

could you bypass it and used all RAM directly

No. Chrome extensions have no control over memory allocation. In fact, they can't even tell how much memory is available or being used. Probably for security reasons. (Actually, this isn't quite true. If you're running the developer version of Chrome there's an API, but most people don't run that version of chrome. And it's not available at all on Firefox.) Note, you can see how much memory WehToEpub is using in the Chrome Task Manager. See: https://www.howtogeek.com/437681/how-to-use-chromes-built-in-task-manager/ Note, if you open multiple copies of WebToEpub, each one is allocated its own block of memory.

[dteviot]

Looks like I'm wrong again. At least on Chrome I can find out how much heap memory is (a) being used and (b) is available. https://developer.mozilla.org/en-US/docs/Web/API/Performance/memory

[dteviot]

Unfortunately, Performance.Memory is pretty much useless for my purposes. From the code https://bugs.webkit.org/attachment.cgi?id=154876&action=prettypatch

Web pages can only learn new data every 20 minutes

My initial thought is I'd need to sample:

1. Before reading any pages.
2. While collecting pages. Say every 10-20 pages.
3. Prior to building the zip.
4. After building zip, but before creating the dataURI

[dteviot]

There's a new API that might be useful performance.measureMemory(). Should investigate further. https://github.com/WICG/performance-measure-memory https://web.dev/monitor-total-page-memory-usage/

[Dimava]

I think there is one and only one thing that can reduce memory consumption in HTML Removing the required node from the document (with el.remove()) (then the #document will be completely garbaged if not keeped elsewhere) well, or cloning the required node with el.cloneNode(true) and garbaging document yet again

[dteviot]

@Dimava
It's more an issue of the extension not being aware that it's running out of memory, so can halt with a useful message. Instead, it just crashes, with no warning to user. I've done some changes to try and reduce the memory consumption. But it's not that easy as the need to fixup links between chapters (there's a handful of sites that have them) means all pages need to be collected and processed. Can't just grab each chapter/web page and stuff them into the epub as you go.