Play nice with Web Assembly

[alex]

This is a big pony (🐴 ), but it'd be amazing if magic-wormhole.rs worked well under Web Assembly.

My reason for wanting this is that I've long wanted to package magic-wormhole up as a browser extension, but my desire to write JS is nil. I've often joked about wanting to do a magic-wormhole Rust implementation and use WASM, so the fact that you're working on this is amzing!

[warner]

WASM is great. Yeah, maybe we write the "wormhole core" approach (in #1) in a way that can be compiled to wasm, and then the IO wrapper gets written in JS (since wasm is a purely computational environment, and has no IO except exchanging messages with its host).

[warner]

Mind you, the amount of JS you'd need to write for the platform-specific IO layer might be considerable..

[warner]

Does this require nostd? Or providing a memory allocator of some sort?

[warner]

I started experimenting with this over the weekend. Two stumbling blocks I've hit so far:

• we use sodiumoxide, and of course that must compile a large C library, which fails under WASM. We're only using it for SecretBox, so I think we can replace this with the libsodium-compatible XChaCha20Poly1305 support that was recently added to the chacha20poly1305 crate (under the RustCrypto AEADs umbrella). #58 is the place for this.
• the magic-wormhole -> spake2 -> curve25519-dalek -> clear_on_drop dependency pathway gets us a clear_on_drop crate that uses a C file to do some assembly-language tricks to zeroize dropped secret-key buffers. It has a feature flag (no_cc) to disable this, but it's not our direct dependency (yet: #63). So we'd need a way to ask spake2 to set a flag that asks curve25519-dalek to set a flag that asks clear_on_drop to avoid the C compile.

I don't know what other problems we'll hit. I was able to bypass the first one by just replacing all our symmetric crypto with no-ops, but I don't know how to get past the second one. Maybe there's some Cargo.toml/Cargo.lock wizardry that lets you set features for a transitive-dependency? Or maybe I should check out local copies of the whole dependency chain, and modify all of their Cargo.toml's to use path= dependencies on everything, so I can edit it all in-place.

[warner]

Once we get the core library compiling to WASM, the next step is to write JS glue code to satisfy its IO needs. The core is written as a function that accepts "events" from the outside world, and returns a list of actions. The application-facing API consists of API events like AllocateCode and Send, and result actions like GotCode and GotMessage. When the core needs IO to be performed, it returns an action like WebSocketOpen, and expects to eventually be given an event like WebSocketConnectionMade and WebSocketMessageReceived:

https://github.com/warner/magic-wormhole.rs/blob/a8b9cc32d9ccaca6a08de9d9f547adc9129f1d53/src/core/api.rs#L169-L186

So the JS glue needs to react to these actions with things like new WebSocket(url) and wiring up the handlers to deliver inbound data to the core. And it needs to peel off the API actions and return them to the JS "application", maybe through an EventEmitter or something.

We don't have any Transit code in magic-wormhole.rs yet, so we're limited to text-message transfers, and custom applications (i.e. a different APPID) that are content with small store-and-forward payloads. I've started poking at the Dilation work that would enable bulk transfers, but I need to finish some of the Python-side code (backwards compatibility with the Transit-based clients) before I'm likely to make too much progress on the Rust equivalent. (I don't intend to implement Transit in magic-wormhole.rs, but of course it'll be a lot easier to get away with that once I've shipped a python version that includes Dilation).

Any thoughts on what the JS API ought to be? My instinct is to make it look like the python Deferred API, where you create a Wormhole object and call methods on it which return Promises. But maybe something that takes .on('message', cb) would be more idiomatic? Or, I guess we could do both?

[warner]

Ok the second problem will be solved by an upcoming release of curve25519-dalek that switches to use zeroize instead of clear_on_drop (which depends upon zeroize-1.0 being finalized, but that sounds like it may happen soon). So we can just wait for that release.

[warner]

@burdges pointed out that cargo features are additive, so if the glue code declares its own dependency upon clear_on_drop = { .. features = ["no_cc"] }, then that'll push the no_cc all the way down the dependency graph. With that trick (and a change to magic-wormhole.rs to replace rustc-serialize with hex), I'm able to get the glue code to compile.

[piegamesde]

The sodiumoxide issue is resolved now, can anyone knowing and interested in WASM give this a second try?

[corbinu]

@piegamesde I would be happy to take a run at it. I have been thinking especially since I love magic-wormhole. I would be interesting to have a static site up that was a WASM version for my non-cli friends and family. Would you be interested in that being a direct part of the project or should I just test as a library and possibly build it separately?

[piegamesde]

@corbinu Great! Yeah, we're all dreaming of this :)

Would you be interested in that being a direct part of the project or should I just test as a library and possibly build it separately?

Tough question. I suggest you to try to include Wormhole as a library first, and if this doesn't work, we'll can talk about integrating it into the repository.

[andipabst]

I wanted to poke around this a little since I'd really love to use magic wormhole on the go in a browser (since a browser is just ubiquitous these days). I don't have any problem with writing the JavaScript wrapper/interop, but I don't really have much Rust experience.

So I tried to include magic-wormhole as a library, as you suggested @piegamesde.

magic-wormhole = { git = "https://github.com/magic-wormhole/magic-wormhole.rs", tag = "0.2.0"}

But unfortunately I can't get it to build as wasm. I could get around missing C-libraries by adjusting the build target wasm32-unknown-emscripten or wasm32-wasi. But now I am stuck since some libraries (e.g. socket2 for websocket) just are not available for wasm. That of course makes sense, since the browser/javascript would handle the websocket connections as part of the IO wrapper around magic-wormhole.rs

I have created a minimal example repository based on the wasm-pack-template for reference.

Could you please point me in the right direction on how to include/build this for wasm as a library?

[piegamesde]

@andipabst thanks for looking into this. First of all, please use some recent commit instead of a release. The last release has been a while ago (sorry about that).

As far as I can tell socket2 is not part of our WebSockets stack. (The latter may be problematic as well, but one issue at a time.) We use socket2 for finding a direct connection. For WASM, that feature must be hidden behind a cargo feature flag. (Feel free to just rip it out for now to get things working.) There are some transitive dependencies of socket2 too (namely, async-io because of async-std). I don't really know how to fix that one yet.

[andipabst]

@piegamesde Thanks for your support! I had a look at it and could get rid of most errors by either ripping the code out or using cargo feature flags, as you suggested.

Now the only thing left is the direct TCP Connection in transit.rs. How should this be approached? Maybe this could be implemented using Websockets and I also thought about using WebRTC for the communication. But I think in the end the transit relay might have to be adapted for this (e.g. to not only handle tcp connections, but also websocket connections). Or do you have another idea or concept?

[piegamesde]

Now the only thing left is the direct TCP Connection in transit.rs. […] in the end the transit relay might have to be adapted for this

Indeed. We are in the process of extending the relay protocol to support WebSockets connections in addition to plain TCP. Some links that may help you with this:

• https://github.com/magic-wormhole/magic-wormhole-protocols/pull/16
• https://github.com/magic-wormhole/magic-wormhole-protocols/pull/10
• https://github.com/psanford/wormhole-william/pull/63
• https://github.com/magic-wormhole/magic-wormhole-transit-relay/pull/23

For a start, I suggest that you simply hardcode the default relay server and ignore the rest, as we already know that it supports WebSockets. Once it works, we can talk about implementing the necessary protocol extensions (I'd be willing to help with that).

[piegamesde]

@andipabst I see that you have some commits on your fork. Can you please take the changes that prepare the repository for WASM support (dependencies, feature flags, code gates) and PR them?

[andipabst]

Hi @piegamesde, I tried again for some time, but unfortunately I was not able to get it working completely. I have pushed my work in progress state in my fork and in a separate repository for the webapp.

Should you decide to have a look at it and get it working, I would be happy to assist with the web part! But my Rust skills are simply not enough to get that part running at the moment.

[piegamesde]

I started copying some of your code over, please check out what I've done in the dev branch and let me know what you think.

Most notably, I try to go without a wasm feature and do everything based on target platform instead. Conversely, even when compiling against WebAssembly you have the choice how many features of the protocol you want to support. (In the current commit, this is just the core protocol, but in the end we want to support everything but the port forwarding feature.)

I'll go ahead and try to integrate your transit changes next. Note that I haven't compiled against wasm target yet so that part is entirely untested. I'll also have to figure out a story how to build everything in CI.

[andipabst]

I had a look and it looks really good! Basing it on the target platform is a good idea, since I really had some trouble with the feature flag. FYI, wasm-pack is building the package against the target wasm32-unknown-unknown.

I think it should be possible to include a cargo build --target wasm32-unknown-unknown into the CI pipeline, shouldn`t it?

[piegamesde]

Thanks! Actually I'm not that happy about all the cfg flags I simply spammed in the transit module, but it will have to do for now. I'm pretty certain that the current transit handshake does not work and needs to be adapted for WebSockets, but that shouldn't be too hard.

I think it should be possible to include a cargo build --target wasm32-unknown-unknown into the CI pipeline, shouldn`t it?

Yes, that's the plan

[piegamesde]

I think the changes should now be ready for actually trying them out in a browser. Since I've never done any web development beyond most basic HTML+CSS, I fear I won't be able to help out much there.

As far as I understand, one of the next steps would be to provide a JavaScript interop API, because most websites are JS based. Let me know if I can help anything there

[andipabst]

Thanks for your changes! I implemented a basic wasm integration. I had to change some parts, since std::time::Instant::now() is not implemented for wasm, but it compiles and runs now until it reaches the handshake.

Then I run into an issue, as you predicted: The handshake via the websocket relay server fails since it opens a new websocket connection to the relay server which then gets a "welcome" response instead of the expected handshake messages.

[piegamesde]

Actually, the handshake should work fine, even though I haven't really tested it yet. What you describes sound like you are trying to connect to the rendezvous server as relay. I'm not sure if the public relay server even has a WebSocket endpoint. You can use my test server though, it should be reachable as tcp://piegames.de:4001 and ws://piegames.de:4002

[andipabst]

Thanks for the hint, that was indeed the issue :facepalm:

I could use with your test server for developing on my machine, but for making a hosted version available I needed a secure websocket endpoint (wss://), so I had to dust off my old Hetzner server a bit and run my own relay server (behind a nginx proxy for letsencrypt ssl termination).

You can see the result here: https://magic-wormhole-wasm.netlify.app/ It is still very rough, but it's a proof of concept and it works :smile: :tada:

So going forward, I will polish the code a bit more and then plan to separate the wasm package and my reference implementation of the web-app using it.

[piegamesde]

Oh I forgot to add the TLS reverse proxy to the endpoint, hence no WSS. To be honest though, I'm not sure that using TLS provides any additional security given that the transit protocol is built to work with plain TCP connections too. But I'll see if I can add it anyways, shouldn't hurt either.

[andipabst]

No worries. I totally agree with you, but when serving the website over https (which should definitely be done), browsers also require using secure websocket connections. (See e.g. https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_client_applications#security_considerations)

[piegamesde]

Thanks for explaining. wss://piegames.de/wormhole-transit-relay should now work too

[andipabst]

Thanks!

As mentioned above I polished and documented the code a bit and copied it over to include it in this repo, see #163. I also adapted the GitHub Actions files so the wasm-assets are built and packaged.

The next step would be to publish this artifact to the JavaScript package registry npm. See the wasm-pack documentation for more details on the technical process.

@piegamesde Can you do please do this, as you are a maintainer and also publishing the rust parts to crates.io?

Meanwhile, I would go on writing the JavaScript/TypeScript reference implementation I mentioned above.

[piegamesde]

This is pretty cool, many thanks! However, I must admit that I have little knowledge about JavaScript and the npm ecosystem, so I wouldn't feel comfortable with pushing things there at the moment. Would you mind becoming the maintainer for the JavaScript bindings of wormhole-rs?

[Sky-NiniKo]

Edit

I found the error and it was very simple. I am stupid. Look at #189 for the fix.

Original post

Hi, I'm a beginner in Rust and even more in Wasm but I tried to make a web application using yew and magic-wormhole.rs and I got this error while trying to upload a file. Logs from the console (F12 -> Console):

panicked at 'Error: getrandom: this target is not supported', /home/niels/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand_core-0.5.1/src/os.rs:63:13

Stack:

getImports/imports.wbg.__wbg_new_abda76e883ba8a5f@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca.js:402:21
console_error_panic_hook::Error::new::h0d0fabec9486fdef@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[14162]:0x4f80b7
console_error_panic_hook::hook_impl::ha02f1ef761b122b6@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[2713]:0x34b3ba
console_error_panic_hook::hook::h77afd9ec18c650a2@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[16028]:0x511213
core::ops::function::Fn::call::ha887bd5a651c1c2d@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[13452]:0x4ed1a2
std::panicking::rust_panic_with_hook::ha2f7b95083ec77f3@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[5452]:0x40c00c
std::panicking::begin_panic_handler::{{closure}}::h47a7e7226e82fb78@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[6812]:0x449861
std::sys_common::backtrace::__rust_end_short_backtrace::h114c7ac12e473e06@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[18384]:0x525920
rust_begin_unwind@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[11258]:0x4c5140
core::panicking::panic_fmt::hae2225a32154cb21@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[15335]:0x5083f4
<rand_core::os::OsRng as rand_core::RngCore>::fill_bytes::he1c5c8da3ce8e69d@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[3098]:0x36edce
curve25519_dalek::scalar::Scalar::random::h3a482c24f807c76f@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[4271]:0x3c77d7
<spake2::ed25519::Ed25519Group as spake2::group::Group>::random_scalar::h5074f6af435f4cfa@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[15136]:0x5058bc
spake2::Spake2<G>::start_symmetric_with_rng::h9310dbd0fe484fee@http://127.0.0.1:8080/yew-app-21e2076cdb23f7ca_bg.wasm:wasm-function[3682]:0x39df73
spake2::Spake2<G>::start_symmetric::h4a14f98f0ef1f101@http://127.0.0.1:8080/yew
...

I think I understood that curve25519_dalek wants to generate a random number but it can't because rand_core doesn't support wasm32-unknown-unknown:

• https://github.com/rust-random/rand/issues/886

I also tried andipabst's code without much success. It doesn't produce the same Traceback ~but I think the problem is the same~. (Edit: After testing the fix, I can say that it is another issue.) I think you don't have much control over this kind of problem, but who knows.

[Sky-NiniKo]

So now that this is settled, I have a new problem. It is likely that this is a misunderstanding of your module on my part and maybe this is not the best place to ask if that is the case.

But, here is the problem: when handshaking with the peer, I get in the console ​`follower_connect` timed out almost instantly after Using secretbox for encryption and then a fatal error.

My code is available here if you want to try: https://github.com/Sky-NiniKo/magic-wormhole-yew