[PROPOSAL] Canvas rendering in a separate thread

[afrou04]

I want to improve the performance of the main thread while using html2canvas.

I need an option that allows the rendering process to be executed in a separate thread using transferControlToOffscreen.

[afrou04]

Is there a way to do this using a web worker?

I'm using a document here, so it can't be processed by a web worker. (document cannot be used because the web worker cannot access the dom)

Is there any good approach?

[ianobermiller]

Most of the time spent is not rendering the canvas, IIRC, but it is building up the clone of the HTML that will be shoved into an SVG that will then be rendered on the canvas. So I don’t know if using a web worker would actually improve anything.

[TimonLukas]

I've looked a bit into this, and I think there is one possible angle that could improve performance and allow offloading a good part of the operation.

As @ianobermiller wrote, the cloning operation is the more expensive part. I haven't tried to measure whether the serialization & reading takes longer, or creating the new elements. There is no way I can think of to outsource the first part to a web worker since they don't have DOM access. And looking through the code, operations like window.getComputedStyle() are probably some of the heaviest in the process.

Building up the clone however can be done in a worker! And, as a nice side effect, this makes rendering in the worker the easier choice. Workers don't natively support these APIs, but Google and the AMP project have created an implementation that seems to at least be sufficient for creating the clone.

I quickly did a couple of performance recordings of the main website of this library, and they all look pretty consistently like this:

The left part is the cloning, the right part is the rendering of the canvas. I'm not sure how I could cleanly differentiate between the reading and writing parts of the cloning process, but it looks to me like there is a lot of potential for moving some of the blocking work off the main thread.

I'll try to look into this a bit. If anybody with more experience in web workers would like to support feel free to reach out, not sure how far I'll come. But if we can make use of multiple threads, it could open up very cool applications! I've been thinking about building a small generator for videos & gifs of changes being applied to code in the browser, and my current implementation would rely on html2canvas to generate the images, which would then be streamed to a running ffmpeg.wasm process to create the finished output. Currently it wouldn't really be feasible to generate a 10 second 60 fps gif with this, as every frame would take ~2 seconds to render (according to my measurement of the main page). If I could instead change the DOM a bit and have a callback raised once the serialized information for the worker DOM has been transferred, I could potentially generate 10 frames or more in parallel, which makes it at least kinda feasible!

---

I'll document my findings here, but I'll put them behind a spoiler so I don't increase the thread size even more :)

Possible approach

Looking into the source code of worker-dom (especially src/main-thread/install.ts, we'd have to do three main steps:

1. Build a new file dist/worker.js which will be loaded in a worker environment w/ cloned DOM without knowing it
   • It has to export a function adjust that can receive the write commands used in the DocumentCloner
   • It has to export a function which wraps the actual rendering process, which shouldn't have to change at all - we can pass the WorkerDOM to the parser and an OffscreenCanvas to the renderer. It will just have to return the finished image instead of the canvas itself.
2. Create proxy code for the worker
   • Instead of cloning the document and creating the iFrame we can create a DocumentAdjuster it will use the function adjust exported by the worker to apply the write operations in the DocumentCloner that have to read from the real DOM
3. Start the worker using worker-dom, apply the DocumentAdjuster, and return a Promise<ImageData>

I believe this approach should work without changing the current code too much, but it would definitely take quite a bit to write.