bevy icon indicating copy to clipboard operation
bevy copied to clipboard

Published 20 hours ago • verified publisher icon bevyengine

WebAssembly multithreading tracking issue

Open colepoirier opened this issue 2 years ago • 23 comments

UPDATE: This is on hold while TaskPool and Scope are reworked

Motivation

Currently, Bevy can only run single threaded on WebAssembly. Bevy's architecture was carefully designed to enable maximal parallelism so that it can utilize all cores available on a system. As of about six months ago the stable versions of all browsers have released the web platform features needed to accomplish this (SharedArrayBuffer and related CORS security functionality). I think now is a good time to attempt to make Bevy run natively in the browser like it does on the desktop: fully multithreaded.

There are three distinct tasks that will enable the accomplishment of this goal:

  1. Create a modified versions of task_pool::{Scope, TaskPool, TaskPoolBuilder} that run on wasm called wasm_task_pool::{Scope, TaskPool, TaskPoolBuilder}, and use those instead of the single_threaded_task_pool::{Scope, TaskPool, TaskPoolBuilder} (TODO: create issue and link here)
  2. Modify bevy_audio so that it runs mutlithreaded in the background (not on the main thread) on wasm (TODO: create issue and link here) a. Contribute the functionality needed to to do background multithreaded audio using WebAssembly via the web platform's AudioWorklet API to the upstream dependencies of bevy_audio (TODO: create issue and link here): i. cpal (TODO: create issue and link here) ii. rodio (TODO: create issue and link here) iii. NOTE: as outlined below in the "Insights provided by developers who have tried to make things that run multithreaded on wasm" section, it may be necessary to PR wasm-bindgen to solve this issue https://github.com/rustwasm/wasm-bindgen/issues/2367 b. Make necessary changes to bevy_audio to make use of this added upstream functionality (TODO: create issue and link here)
  3. Modify bevy_ecs, bevy_render, and the rest of bevy to run multithreaded on wasm (TODO: create issue and link here) a. Insights from @alice-i-cecile below. "For the ECS, there are two relevant bits:" i. Our multi-threaded executor ii. Parallel iteration over queries

NOTE: as outlined below in the "Insights provided by developers who have tried to make things that run multithreaded on wasm" section, if we need to do something where we cannot use wasm-bindgen we will need to manually set the stack pointer in our code because this is one of the things wasm-bindgen does. @kettle11 has put this functionality into a tiny crate: https://github.com/kettle11/wasm_set_stack_pointer

Background on SharedArrayBuffer

There is a good reason Bevy, and many of the existing projects that run on wasm only run single-threaded. Shortly after the time of the initial introduction of the SharedArrayBuffer web API - which would allow true unix-like pthread-style mutlithreading using wasm in the browser - the Spectre exploit was discovered.

Due to SharedArrayBuffer being a wrapper around shared memory, it was a particularly large vector for Spectre-style exploitation. In order to maintain their strong sandboxing security model, browsers decided to disable the feature while a proper solution was developed. Unfortunately, this eliminated the necessary functionality to allow true multithreading on wasm. What existed in the interim was a much slower emulation of threads using WebWorker message passing.

Thankfully, as of about six months ago all browsers have re-enabled a redesigned and secure version of SharedArrayBuffer. According to this article on the chrome development blog "Using WebAssembly threads from C, C++ and Rust" https://web.dev/webassembly-threads/, true pthread-style multithreading is now possible on wasm in all browsers, with the small corollary that users may need to write a small specialized javascript stub to get it working exactly in the manner they need. Given that it has been stable for this long, and that some chrome developers have even published a github repository with an implementation for this for rayon using wasm-bindgen, I think now is a good time to investigate how to make Bevy run natively in the browser like it does on the desktop, and try implementing this to see if it will actually work.

Insights provided by developers who have tried to make things that run multithreaded on wasm

@kettle11 provided some good insights into quirks and solutions to multithreaded wasm on discord here on 19 November 2021:

""" In the past I got AudioWorklet based audio working with multithreaded Rust on web. It's certainly possible.

When working with wasm-bindgen it requires some messy code because wasm-bindgen uses the Javascript API TextDecoder which isn't supported on AudioWorklet threads. The way I got around that is by not using wasm-bindgen on the AudioWorklet thread, but that requires a few hacks:

Scanning the Wasm module imports and importing stub functions that do nothing for every Wasm-bindgen import. This is OK because the audio thread can be made to be pretty simple and avoid doing direct wasm-bindgen calls.

Allocating a stack and thread local storage for the worker. wasm-bindgen's entry-point does this normally, but wasm-bindgen's entry point also calls the main function which we don't want for the AudioWorklet thread. So we need to use our own entry point and manually set up the stack / thread local storage.

I opened a wasm-bindgen issue about theTextDecoder thing about a year ago: https://github.com/rustwasm/wasm-bindgen/issues/2367

Also wasm-bindgen solves the "how do we set the stack pointer?" issue by preprocessing the Wasm binary and inserting the stack allocation code, but I found a way to do it without that which I put together into a tiny crate: https://github.com/kettle11/wasm_set_stack_pointer """

Resources

colepoirier avatar Mar 02 '22 02:03 colepoirier