Discussion and development of web/wasm version

[falkTX]

For those unaware, https://cardinal.kx.studio/ is totally a thing now for some reason. Also https://minicardinal.kx.studio/ for a mini/stripped-down version that loads faster.

There are some issues and things still to do. Because this is for playground and testing and fun, and not all that serious, let's not open more than 1 ticket for web related things. Everything web/wasm related should be on this ticket.

Stuff to do or implement:

• [x] audio input
• [ ] touch events
• [x] clipboard (do fake internal one if not possible to expose in browser)
• [x] webmidi
• [ ] host time controls (on menu bar, providing fake but valid BBT values)
• [x] intro message explaining what it all is, what to expect etc
• [x] detect incompatible browser setups and refuse to even load (noscript, no simd, no webgl2)
• [x] build libsndfile and fftw for extra module features
• [ ] minimum gui size (CSS is enough?)
• [x] allow url parameters to tweak page behaviour, also load patches as base64 encoded string
• [x] test loading all modules and disable those with custom non-GLES2 compat code

Known issues:

• [x] Loading zst compressed project file fails to load patch.json
• [x] opening salamonis-mono-regen example patch triggers an exception
• [x] going full screen can mess up with the viewport size and coordinates
• [x] ctrl+a clears all text
• [x] audiofile player is missing regular audio support (only mp3 works)
• [x] audiofile and midifile inside ildaeil fail to show file dialogs
• [x] latest changes made performance in chrome worse?? firefox still fine..
• [x] compressed data assets > 50Mb
• [x] missing versioning to js, wasm and data downloads (so they are updated automatically when reopening the page as needed)

Stuff to fix or workaround:

• [x] apply high-dpi event offset
• [x] test and implement fractional scaling "high-dpi" (only supports 2x and 3x now)
• [x] prevent ctrl/cmd+ shortcut keys from going to the browser
• [x] prevent browser from zooming the page
• [x] oddities with mouse down and leaving browser window

Undecided:

• [ ] host parameters
• [ ] host cv
• [x] persistent storage (client-side browser DB)
• [x] bundle a few simple LV2 plugins (mda, caps, tap, etc)

[falkTX]

Leaving a note to myself, https://github.com/Dreamtonics/juce_emscripten/blob/master/modules/juce_audio_devices/native/emscripten/RtMidi.cpp has rough but working code for webmidi. Not that it is complicated to do, but helps to have something to compare against.

[falkTX]

high-dpi screen should work now. tested locally with a mac mini.

if you have such a screen, let me know how it works!

[john-the-bard]

The ping pong delay from MSC makes a ton of noise if you load it, this basic patch which should play a single plucked note every second into a short and feedback heavy delay.

NOISE WARNING the output is connected and it immediately clips.

NOISEWARNING_Pluck.tar.gz

[diimdeep]

Insanity. Wowzers. Would be cool to load patch from url encoded parameter, https://cardinal.kx.studio/?patch=encoded_blob like this

[falkTX]

Insanity. Wowzers. Would be cool to load patch from url encoded parameter, https://cardinal.kx.studio/?patch=encoded_blob like this

I thought about this, and while I typically dislike such approaches, but I think it makes sense here. A way to generate such link directly from the page would also be quite cool, so we can:

1. open cardinal web page
2. make a patch showing a specific thing
3. use file -> generate pasteable link (or whatever better name there is)
4. send that link to someone for quickly trying the patch

there are 2 things though that need to be done before we can tackle this:

1. fix opening zst compressed patches
2. allow for versioned caching of wasm data, so page reloads are instant (loading 100Mb of resources everytime to check new patches is just painful)

will add to the todo, thanks for the idea!

[pauldavisthefirst]

drag doesn't work to scroll boundaries of view area. click on output "socket", drag upward to reach module that is currently "off-screen" .. drag will break.

[pauldavisthefirst]

if scroll is used to zoom, easy to get to a point where cardinal does not occupy entire browser screen, at which point the coordinate determination seems to be messed up. drags will affect things not under the mouse.

[falkTX]

drag doesn't work to scroll boundaries of view area. click on output "socket", drag upward to reach module that is currently "off-screen" .. drag will break.

mentioned on the 1st post as

oddities with mouse down and leaving browser window

and

if scroll is used to zoom, easy to get to a point where cardinal does not occupy entire browser screen, at which point the coordinate determination seems to be messed up. drags will affect things not under the mouse.

known issue, I fixed it already locally but didnt deploy a build for it yet

[falkTX]

Small update: fixed up the ildaeil audiofile and midifile plugins, adjusted for more high-dpi screens, and updated view menu to follow official Rack 2.1.2 changes.

With the ildaeil fixes it is now possible to get some MIDI going. Follow these steps:

1. Add ildaeil module
2. Add ildaeil expander midi output, place it touching the right side of ildaeil
3. In ildaeil, switch to internal plugins (it is JSFX by default)
4. Load "MIDI File"
5. Set "Host sync" to off, "Repeat Mode" to on
6. Press "show custom gui" button
7. Pick a MIDI file from your PC

Right-click the expander module for polyphony options.

[falkTX]

Good news, the issue with loading compressed project files is fixed. (or better said, a workaround is in place) Somehow zstd extracts the tar with incorrect permissions, and then fails to load. We now go for 777 for any file to be open, and all is good. The "files" are all virtual anyway, so doesn't matter on the web context.

Also, you can now pass it a patchstorage "slug" to the URL as a way to load that patch on start. For example https://cardinal.kx.studio/?patchstorage=fern-sequence and https://cardinal.kx.studio/?patchstorage=example-patch-2 The URL parameter is the same that is used on patchstorage.com views, e.g. https://patchstorage.com/example-patch-2/

[falkTX]

Another update, this is fun :laughing:

Thanks to previous work done in https://github.com/DISTRHO/PawPaw I could just repurpose it to build generic opensource libs for wasm. For now that is libogg, libvorbis, flac, opus, sndfile, liblo, fftw and lv2 related libs. This helps a lot on building opensource plugins in wasm. Also these extra libs allow to get the extra/proper file support in the audiofile module and brings in the audio-to-cv-pitch module (which required fftw)

I did a few more tweaks and made Cardinal able to load audio plugins from the "filesystem" (JSFX dont really count, they are not binaries). I have compiled artyfx, die-plugins (imported ardour fx), fomp and mda LV2 plugins, and added them to the site/webapp. Just open up ildaeil and they will be there for the LV2 option.

[falkTX]

ok a couple more things now:

• fixed up event key input handling a bit, ctrl+a no longer clears all text
• keyboard is no longer propagated to browser, so stuff like ctrl + scroll works now (some key combos like ctrl+n and ctrl+t cannot be overridden)
• pointer lock is implemented and enabled by default (opt-out in view menu)
• added full screen menu option

[dromer]

Going out of full screen returns to the wrong size render (x/y mouse coordinates are still correct, but don't align with the view).

[falkTX]

Going out of full screen returns to the wrong size render (x/y mouse coordinates are still correct, but don't align with the view).

I have some ideas for this, the fullscreen mode typically changes the viewport to render in "pixel perfect" mode by default, while typically contents are scaled instead. When entering fullscreen, we can request how to deal with the viewport and keep things as-is. Will try later

[falkTX]

And more news for the web side, Web MIDI is implemented! Both MIDI input and MIDI output. Does not work on Firefox, as it is not implemented there. Only Chrome/ium-based browsers work for that. And if you are using an Ubuntu based system with snaps, alsa-midi access is disabled there so this wont work.

The web audio stuff was rewritten from scratch too, I was using SDL before but it was doing some weird stuff behind the scenes, specially for audio input. Oh yeah, I got audio input to work too, but did not enable yet as the permission request for it would make the page look too suspicious. ("Why does this random page want mic access?") A button on the menubar to request mic access might be the best, to handle later. I did try the web audio input, and quality is really awful, so we are not losing much.

For the curious ones, relevant code for this web audio + midi is at https://github.com/DISTRHO/DPF/blob/develop/distrho/src/jackbridge/WebBridge.hpp This is not using worklets yet, as that has special security considerations and just a pain to setup. With the current single-threaded approach we do not get the best performance, but in turn it is also possible to just run the html files directly in any compatible browser.

I will try to have the html/js/wasm files built as part of 22.07 release on CI. There is nothing special about it, just emscripten with USING_GLES=true build flag.

[falkTX]

Audio input is now enabled. And MIDI is disabled by default, as I try to optimize things a little big (chrome is behaviour worse after changing to custom written code vs old SDL2 usage, for some reason)

You can enable audio input and MIDI under the engine menu.

Audio input quality is really subpar here.. and forces itself as mono even if we request 2 channels. You can see the blocky-ness of it in a spectrum, really ugly :(

But, well, feature is enabled now.

[falkTX]

A bit more news on the input and processing.. For one, the issue I had with quality seems related to pulseaudio. using chrome with alsa the input quality is much better.

Chrome cannot do stereo input though, only Firefox does.

Also "fixed" the choppy audio in chrome after the latest updates, turns out it is just tweaking buffer size. SDL was likely buffering things, not using a real 512 samples as requested. Default now changed to be 2048 samples, but you can change this in the engine menu. Might help make the webapp work on slower machines.

[falkTX]

There is a mini version now at https://minicardinal.kx.studio/ Contains only core and fundamental modules, extras like carla/ildaeil are also not in there.

The mini version assets have a combined 15Mb size. It is small enough that Firefox is happy to cache the page, so that's nice :)

Feel free to use and share!

[falkTX]

A few more things:

touch events are now enabled, sorta.. because neither Rack or pugl can deal with touch events, they are converted to mouse events in hopefully a way that makes sense.

• only 1 finger used at the moment.
• quick press does a mouse click
• press and hold does mouse right click
• press and move does a mouse click + drag (so if you dont want to trigger a right click menu, move finger after pressing)

moving around the canvas is still an issue. I am considering mapping 2 finger movements to mouse wheel, will need to run some tests.. I managed to make it load on my phone, but since it is a > 7 year old one, its performance is quite terrible. otherwise works. testing and feedback is appreciated.

the hosted files are now gzip compressed, which makes them smaller and thus faster to load. we can gzip compress the files and host both compressed and uncompressed, then apply some apache magic to load compressed files if browser supports it. it is a nice reduction in size!

I will push the setup for this later.

A final thing added was detection of required browser features before trying to load the web assembly program. Old browsers cant load Cardinal web, but that is kinda expected, it is nice to have a list of unsupported features.

And speaking of unsupported browsers, with this we now know why it does not work on Safari - lack of wasm simd support. That means it wont work on any iOS device at the moment too.. Apple seems to be pushing in wasm territory though, so perhaps this will be coming soon. https://platform.uno/blog/the-state-of-webassembly-2021-and-2022/ seems to give a good overview of the current situation.

[falkTX]

Oh, and another small change I forgot to mention: when browser tab/window is not visible, cardinal wont spend time rendering the webgl gui anymore. which makes it use less cpu when running in the background.

[falkTX]

Good news, with more optimizations and compression, the wasm binary and data downloads are now below <50Mb.

This is great because, at least in Firefox, downloads with less than 50Mb are allowed to be cached.

Additionally I did some tests for clipboard support. While it is possible to support it, and I verified that it actually works, mapping it to current APIs as used by DPF and Rack requires us to use asyncify which complicates the build and makes it slower. in the end it seems not worth it to pursue. note that this is only for pasting from system clipboard into the site/webapp, the other way around does not have such strong limitations. we can even fake/force copying out of the webapp by creating a hidden html element, setting its text to the one we want and instruct the browser to copy from it.

so starting today we have a sorta fake clipboard in the webapp, with it having internal data for dealing with copy&paste and allowing to copy out of the webapp as an extra.

[falkTX]

We have URL parameters handling now!

Both passing full patch, or a link to some file online.

Use patch keyword for patch file in URL, alike https://cardinal.kx.studio/?patch=eyJtb2R1bGVzIjpbeyJpZCI6IDEsInBsdWdpbiI6IkNhcmRpbmFsIiwibW9kZWwiOiJUZXh0RWRpdG9yIiwiZGF0YSI6eyJldGV4dCI6IldoYXQgYSBoYWNrIHdvdyJ9fV0sImNhYmxlcyI6W119Cg==

Use patchurl for loading an online patch file, alike https://cardinal.kx.studio/?patchurl=https://patchstorage.com/wp-content/uploads/2022/07/JTB-wasm_house_toshare.vcv

The embed patch in the URL is kinda disappointing, even with zst compressed data we easily get into the limits of how long we can make the URL. It works, but it is very limited.. It always uses base64 encoded string, if there is a better way to pass binary data as string please tell me.

The remote file seems the best approach here. you can host a file on a pastebin like service, and pass the raw contents url as parameter to cardinal web app. Can be quite useful to quickly show some patch to someone.

Have fun!

[Ulfmusic]

Hi Could I please have some simple instructions on locally hosting the wasm version of Cardinal? Thanks for a great DAW

[falkTX]

Download the artifacts and serve them under a web server. There is no magic needed, these are just static files. Either grab the 22.07 release files, or the latest build from the actions tab.

[aolney]

In a word, amazing :partying_face:

I'm looking at incorporating this into a tutorial website/book I'm working on. Some questions with that in mind:

• Looks like computer keyboard for midi input was not ported over. Staying in-browser and ideally within Cardinal, is there an option besides the Impromptu 12-key module? This type of capability is really useful for beginner patches.
• Can menu-based settings be set from the outside? It would be great to
  • Detect Chrome and automatically turn off Lock cursor while dragging to avoid "Press esc to show your cursor" messages
  • Possibly change scrolling behavior for mobile phones (not getting a bottom scroll bar on Android currently)
  • Possibly change the buffer size for mobile phones to something larger than the default

[falkTX]

Cardinal does not do midi-like input via keyboard, never did. For such changes, you can fork the repo and do the changes as you wish.

[falkTX]

Small update: The main https://cardinal.kx.studio/ has been updated to the latest code. Sadly this brings the size of the compressed assets above 50Mb again, and thus they are not being cached by the browser anymore. Files are also not being automatically updated (their cache invalidated) so that is a small thing to fix too

[aolney]

Thanks - going back in the build history, it looks like wasm builds were commented out in build 1188.

The last successful build before that was build 1182. Would this build be the best build for wasm that is still cache friendly?

FWIW I currently have build 1153 deployed for testing/evaluation. I'm deploying through GitHub Pages, so caching is helping me avoid going over their bandwidth quota.

[falkTX]

wasm builds are enabled already. they were failing due to new modules having a lot more code which made the build machine go out of RAM. I have reverted voxglitch to the same version as used in 22.07 (with minor fixes), but we are at the limit of what the free github CI VMs can do..

[aolney]

I think we're missing a file - patchurl.php seems to be required to load a patch from patch storage.