minifb native runtime has inconsistent framerate on macOS

[VyrCossont]

System: I'm on a 2019 Intel Core i9 MacBook Pro running macOS 12.5.1 Monterey.

Context: I'm experimenting with a tool (pico8_rom_tool) to translate a subset of PICO-8 music to Rust code for playback in WASM-4.

This test WASM-4 cartridge (zipped .wasm) (source) currently plays the treble part of the first few bars of Ode to Joy, extracted from this PICO-8 cartridge with pico8_rom_tool:

It does not play properly in the currently shipping macOS minifb native runtime for WASM-4. It sounds like the frame timings are very slow, by multiple frames, and as a result, the music is slow, unpredictably timed, and thus almost unrecognizable. This mostly goes away if the runtime's window is clicked at least once, at which point it seems to run at normal speed.

I've captured audio from each of PICO-8, the minifb native runtime without clicking the window, the minifb native runtime with clicking the window, Firefox using the web runtime, and a resurrected glfw build of the native runtime, and visualized them in Audacity:

Here I've zoomed in on the first few notes. (Note that Firefox won't play any audio until the player is clicked, so the first note is missing from that version.)

The minifb native runtime (after clicking) sounds nearly as good as PICO-8. The glfw native runtime is close but accumulates some drift after a few notes. The web runtime in Firefox is worse than either native runtime (note the wider spacing between notes) but recognizable. I've also tried it in Safari, which is not included in the visualization, but sounds roughly equivalent to Firefox.

I have not yet tried this experiment on the native runtime on other platforms. I'll look at Windows after filing this bug.

Summary:

1. There's clearly something wrong with the macOS minifb build where it's not running at full speed when the runtime starts. I don't have an idea what's causing this; suggestions are welcome.
2. In general, WASM-4 has no mechanism for timing music or sound effects that is independent of counting update() calls. Music and SFX are extremely sensitive to small delays in frame timing. Adding an elapsed-real-time API independent of update() would improve the developer and user experience for anything related to audio, and probably also in general.

[JerwuQu]

On the second point there, switching the WASM-4 APU over from internally being wall-clock based to being tick-based would both make more sense and fix the issues with gaps between tones.

I had an issue open for more precise timing, but closed it due to suspected complexity #472.

I think the minifb issue is probably with minifb itself, so might be good to open an issue upstream.

[VyrCossont]

1. Quite possibly. I'll try to reduce the issue to a minifb-only test case, and if I can do that, I'll close this out and open a bug with upstream.
2. #472 is a great writeup of the issues; thanks for that. After reading that and some discussion in the Discord, I agree that elapsed-real-time is probably not going to solve this, especially the effectively restricted BPMs.

[VyrCossont]

I should probably add a note here on Impact: point 1 only affects attract modes/title screens, and goes away when the user starts interacting with the cartridge.

[VyrCossont]

Just confirmed that this does not affect the Windows native runtime, and also that sound is not a factor, just a symptom, as the Mac native runtime develops framerate issues regardless of whether sound is playing. This seems like a minifb issue, as suggested by @JerwuQu; maybe the currently open emoon/minifb#62.

That said, @aduros, would it be a huge maintenance hassle to ship the glfw runtime as the Mac native runtime for the time being, so that w4 run-native doesn't glitch on Macs? The glfw build still works fine if it's uncommented and the deleted vendorized files are restored.

[VyrCossont]

If that's an acceptable workaround, I've put the change of the default macOS native backend in #574.

[VyrCossont]

Sorry, one more note: I've noticed as I work that the minifb macOS runtime may develop framerate issues regardless of whether the user is interacting with it, so the problem doesn't just affect attract modes and noninteractive demos.

[aduros]

Thanks @VyrCossont, it's a little sad we'd have to maintain the glfw backend, but I guess it's worth it until this is fixed upstream. Have you tested on Windows or Linux? I'm wondering if we should switch there too. I think I remember @JerwuQu reporting an issue where on Linux minifb runs slightly faster than 60 FPS.

I originally switched to minifb because the executable sizes were drastically smaller than glfw. Is there anything we can do to trim the glfw size a bit?

[VyrCossont]

I've tested on Windows and not seen any obvious framerate issues with the currently shipping minifb backend. Haven't tried Linux outside of a VM, but didn't see any obvious issues there either.

As for the binary size, on macOS, a release build of wasm4 is 371 KiB, and a release build of wasm4_glfw is 524 KiB. It's bigger, true, but that's not going to make much of a difference in distribution size when we consider that the release build of the macOS w4 wrapper is 41 MiB, so we're not exactly talking about a scenario where WASM-4 for macOS won't fit on a floppy any more.

[VyrCossont]

Might be worth adding framerate instrumentation to all of the native backends too, since that's a feature we have in the web backend. I'll look into it.