Emscripten: use an AudioWorklet?

Seems like the proper way going forward is to use AudioWorklet.

Originally posted by @uyjulian in https://github.com/libsdl-org/SDL/issues/5459#issuecomment-1079967695

Basically we want the audio callback off the main thread, without the overhead of proxying from a pthread. I need to research AudioWorklets, but this seems like the direction the tech is going...?

I approve of this. SDL_emscriptenaudio.c currently uses BaseAudioContext.createScriptProcessor, however it is deprecated and will be replaced by AudioWorklets.

We're way behind on 2.24.0, so I'm going to bump this to the next milestone, since it's not urgent yet.

AudioWorklet is well-supported across browsers:

https://caniuse.com/?search=audioworklet

Documentation on AudioWorklet:

https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet

Been looking at this awhile now, and it's not clear to me how a Worklet, which has no access to the app's address space, is going to be able to call into the app's audio callback. We're still going to need to feed this thing from the main thread, afaict, so it's questionable if there's any benefit to the added complexity this is going to cause, unless BaseAudioContext.createScriptProcessor goes away entirely.

I'm thinking we need to bump this from the milestone, and (unless @Daft-Freak thinks this is something that can be made to work better, using things I don't understand well) maybe we should just close this as WONTFIX.

Hmm, this looks relevant: https://github.com/emscripten-core/emscripten/pull/16449

(TBH I was avoiding this hoping that emscripten would handle the js/wasm <-> C magic at some point)

I've made an initial attempt at this: https://github.com/Daft-Freak/SDL/tree/audio-worklet. Seems to work reasonably well, but requires enabling threads.

In light of @Daft-Freak's work, I'm going to bump this to 3.2.0 and see what the current state of Emscripten threads are; either we can just switch over to this or we can separate this out into an option if threading support is enabled.

(Let's not do this for SDL2 in any case.)

I really would like SDL2 to not need threads in Emscripten (too many game online stores and static hosts doesn't offer the required headers or this isn't well supported when they do). (btw audio worklets are not pthreads, I don't think they by itself require SharedBufferArray)

Saw an interesting commit here: https://github.com/mackron/miniaudio/commit/25885e416339e23338fe4108f5cca20ba53da5fd

Fwiw, the concern isn't the worklet itself, but that the worklet needs to call into arbitrary app code for the SDL audio callback.

We're thinking of removing the callback mechanism in SDL3--so apps just queue data to be played--which would make the need for pthreads here possibly unnecessary...?

Oh no, this needs threads enabled to even compile. (-sAUDIO_WORKLET requires -sWASM_WORKERS... which requires building for workers and then you get compile errors (something about atomics?) without -pthread)

Wouldn't mind being proven wrong here though, thread support is kind-of a pain...

I was wrong, while wasm workers and pthreads can have different APIs in Emscripten, sadly both require SharedBufferArray (https://github.com/emscripten-core/emscripten/issues/16880). So yes, using Audio Worklets will carry the coop coep header burdens.

I'm messing around with this at the moment.

I'm planning to put this in as a separate audio driver, so if SDL is built without pthread support, the driver just won't be built in at all, so it'll choose the original Emscripten audio backend in that case.

Another issue with AudioWorklet is that it requires a secure context.

If queueing audio will be the next thing, I would think using a similar method as Emscripten OpenAL port would be better, as it doesn't require ScriptProcessorNode nor AudioWorkletNode. https://github.com/emscripten-core/emscripten/blob/5ecd2d9d62fd246525d37a4337cfca858b43e3ef/src/library_openal.js

Technically, threading at all needs a secure context, so if you have that, you might as well use AudioWorklets.

SDL will still compile and run without threads, using the existing single-threaded audio backend, and we don't intend to change that in the near future. AudioWorklets are just an optional backend if you're already using threads.

As for AudioBufferSourceNode (the Emscripten OpenAL thing)... it looks more efficient than ScriptProcessorNode. It might be worth exploring for the single threaded backend if we can reasonably queue buffers without gaps...It looks like it has a callback for when a buffer finishes playback, so if we have two of them in flight, we can use the callback to queue a new buffer when one finishes and the other is still playing, if we can get the timing right for scheduling the new buffer's playback.