Compiling and running scryer as a WebAssembly binary?

[rla]

Hello @mthom! Would it be possible to compile and run scryer as a WebAssembly binary? I worked some time ago on SWI-Prolog WebAssembly port but it turned out to be rather difficult due to:

• Clib/POSIX platform expectations (like getting stuck with the Emscripten's broken readdir and other functions);
• Fighting with the build system (it requires intermediate binary to build itselt);
• Awful difficulties with cross-compiling C as usual.

Also, it was quite inconvenient to provide API for JavaScript. It takes hundreds of lines of handwritten code to bind JS functions to low-level FLI, a SWI-Prolog's FFI.

More info: https://github.com/SWI-Prolog/roadmap/issues/43

It seems like Rust has some support for WebAssembly (https://developer.mozilla.org/en-US/docs/WebAssembly/Rust_to_wasm).

What do you think:

• Would it be possible to build scryer as a WebAssembly binary?
• Could scryer be decoupled from filesystem/POSIX/threading/signals requirements and run without filesystem, clib or any syscalls at all?
• Would it be possible to use wasm-bindgen to generate useful API? (It could be something quick-and-dirty like terms serialized as JSON strings at first)

[cduret]

I have compiled scryer without signal (nix package) under windows OS

[tniessen]

Would it be possible to build scryer as a WebAssembly binary?

I did that one or two months ago, based on c342d18f926a5e5778204e35f80e44844482ae45. It seems possible, but it certainly isn't trivial.

Compilation problems

1. The rug dependency doesn't work with WebAssembly. I exchanged a few emails with @tspiteri, and it seemed challenging to cross-compile GMP. Workaround: Remove the rug dependency and change the default features to ["num"], which uses num-rug-adapter.

2. I couldn't get the crossterm crate to work, which is not really a surprise given the lack of a terminal in WebAssembly, but there might be a solution that I missed. Workaround: Remove crossterm and adapt get_single_char in src/prolog/machine/system_calls.rs to use an imported function raw_mode_single_char instead (which needs to be implemented in JavaScript):

   extern "C" {
     pub fn raw_mode_single_char() -> u32;
   }
   
   pub fn get_single_char() -> char {
     unsafe {
       return char::from_u32(raw_mode_single_char()).unwrap();
     }
   }
   

3. ProcessTime won't work in WebAssembly. Workaround: Replace or remove ProcessTime::now() in src/prolog/machine/system_calls.rs.

4. Signal handling doesn't work in WebAssembly. Workaround: Remove signal handling logic in src/main.rs. (Probably not necessary if you are building the project as a library instead of an executable binary.)

Cross-compiling

I don't remember the exact command I used to compile it, but I used rustup's nightly toolchain for the wasm32-wasi target.

Syscalls in the browser

I implemented necessary syscalls to an extent that allowed to load a prolog source file from a virtual in-browser file system, and to see output written to the stdout file descriptor. Apart from that, I left most of the wasi syscalls unimplemented, but someone with more time could probably get most of them to work.

Result

I got this tiny "Hello world" to work eventually:

:- initialization(hello_world).

hello_world :-
    write('Hello, World!'), nl.

Partial screenshot of the web page (with some debugging info):

I didn't try much more after that, this had already taken me many hours and I mostly did it to gain some experience with Rust → WebAssembly.

[rla]

@tniessen, thank you! That's awesome. Despite these issues, it seems like porting scryer would be easier.

• SWI-Prolog wasm version also does not support GMP;
• Unless using threads, terminal input cannot work anyway since the blocking read call would also block the browser event loop including input events to read user input. SWI-Prolog wasm version has exactly the same issue.
• I see that all IO is collected to system_calls.rs module, this also simplifies things.

@tniessen, do you happen to still have your changes somewhere? It would be a nice starting point.

[tniessen]

Unless using threads, terminal input cannot work anyway since the blocking read call would also block the browser event loop including input events to read user input. SWI-Prolog wasm version has exactly the same issue.

@rla It might be possible to avoid this in some browsers with only minimal modifications in scryer-prolog. I wrote synchronous-channel a while back to allow WebAssembly in worker threads to retrieve information (such as terminal input) from the main thread. I did not test the library in browsers, but it should work. So the idea would be for the main thread to write terminal input to a SynchronousChannel, and a worker thread running the WebAssembly code could read it from the SynchronousChannel. The library supports both blocking and non-blocking read and write operations.

do you happen to still have your changes somewhere? It would be a nice starting point.

I'll try to find and commit them, but it might take a few days.

[jacobfriedman]

Very interested in a wasm module with TCP/UDP bindings. I'm trying to build the SWI-Prolog WASM just to see where I can find those calls... any idea if I'll be able to sniff those calls out with a nodeJS WASI wrapper?

[jacobfriedman]

@tniessen when you're ready to dedicate some time to get a WASM going, I'm right here with @rla . I'd like to get an environment up... 'one to bind them all' with prolog - scryer's the most lean implementation I've seen. Please let me know if you're willing to work together :)

[tniessen]

Sorry @jacobfriedman and @rla, I am super busy with university, OSS, and everything else right now, but it's still on my list. I'd love to collaborate on this, and I'll try to find time soon-ish, I hope.

[tniessen]

@jacobfriedman @rla I am trying to get it to work again but recent changes have only made it more difficult.

• crossterm doesn't work, this affects keyboard events, workaround in https://github.com/mthom/scryer-prolog/issues/615#issuecomment-650765495
• hostname doesn't work (not important)
• rustyline doesn't work, this affects essentially everything related to user input
• ring doesn't work (but might with some hacks)
• openssl doesn't work (but might with some hacks)
• native-tls doesn't work
• sodiumoxide doesn't work (but might with some hacks)
• slice-dequeue doesn't work

I am not sure why there are so many different crypto libraries in the list of dependencies.

TCP (the sockets library) won't work in WebAssembly, assuming it's supposed to run in a browser.

[jacobfriedman]

Well, a red flag for me: "I am not sure why there are so many different crypto libraries in the list of dependencies." @mthom is there a workaround?

What I've been seeing is that many libraries just wrap C instead of implementing in the native language. I wonder if these dependencies can be replaced with native (rust) drop-ins.

Are you using emscripten to compile, or compiling straight to WASM?

[triska]

Thank you a lot for resuming work on this project!

The cryptographic crates are used to support various cryptographic predicates in library(crypto), and also TLS connections in library(sockets).

The big bouquet of crates is needed because none of the existing crates covers all the functionality that library(crypto) exposes. Some of these crates are more convenient to use than others, and all of them have unique functionality that only they provide, at least in this simplicity.

If you do not need this functionality, you can, in your WASM branch, simply remove all cryptographic predicates from system_calls.rs, starting from https://github.com/mthom/scryer-prolog/blob/10e92eec32ac043fb9e398482ae20f9921c7eee4/src/machine/system_calls.rs#L4749 up to and including https://github.com/mthom/scryer-prolog/blob/10e92eec32ac043fb9e398482ae20f9921c7eee4/src/machine/system_calls.rs#L5197

and then simply remove the inclusion of these crates. This could be useful to start with this project. If at all possible, it would be great to preserve the cryptographic functionality as far as practical also in the WASM port.

Please let me know if you have any questions about this.

[jacobfriedman]

Thank you for the help! @tniessen will you take the wheel on this, as I'm relocating this week? If so please fork/branch & post the URL here :)

[triska]

One additional remark: If you plan to get one of the cryptographic crates working, I suggest to focus on ring. This is the most useful and well designed of the cryptographic crates, and porting it would enable a lot of the cryptographic functionality, most notably secure hashes, encryption and authentication.

The other crates are used for more rarely used hashes, and low-level reasoning over elliptic curves. I plan to get rid of specifically the openssl dependency as soon as a native Rust crate provides the needed functionality for elliptic curves.

[infogulch]

simply remove all cryptographic predicates from system_calls.rs ...

and then simply remove the inclusion of these crates

I wonder if this would be a good use-case for a cfg flag (crypto perhaps?), enabled by default, that if disabled omits the crypto-related system calls and their dependencies. This way, if the project is being built for WASM the cfg flag could simply be disabled.

[tniessen]

@tniessen will you take the wheel on this, as I'm relocating this week?

Sure, I'll try to get a minimal version to work.

[jacobfriedman]

Cool. I can work on it first thing next week. Also- are you working with emscripten?

On Thu., Apr. 8, 2021, 1:06 p.m. Tobias Nießen, @.***> wrote:

@tniessen https://github.com/tniessen will you take the wheel on this, as I'm relocating this week?

Sure, I'll try to get a minimal version to work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mthom/scryer-prolog/issues/615#issuecomment-815990637, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFKHXWQRJBPYMWT2E5N6NDTHXPBZANCNFSM4OJ3LNXQ .

[tniessen]

@jacobfriedman No, I am trying to get it to work with cargo wasi. Emscripten might come in handy when we need to provide the WASI environment in the browser, but I'd personally prefer a "clean" build without Emscripten first.

[tniessen]

Oh, and best of luck on your move :)

[jacobfriedman]

Thanks!

Ok, good to know. I'll start digging into the docs.

On Thu., Apr. 8, 2021, 1:26 p.m. Tobias Nießen, @.***> wrote:

Oh, and best of luck on your move :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mthom/scryer-prolog/issues/615#issuecomment-816004157, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFKHXT5QLPOHRBGWQQUZB3THXRNNANCNFSM4OJ3LNXQ .

[tniessen]

I made the crypto and terminal dependencies optional, but getting rid of slice-deque seems like a bigger challenge here. That will probably become a large diff.

[jacobfriedman]

There must be a better way than a huge diff. In my past experiences the problems always trickled down over time.

Maintainers, what do you say?

On Fri., Apr. 9, 2021, 9:57 a.m. Tobias Nießen, @.***> wrote:

I made the crypto and terminal dependencies optional, but getting rid of slice-deque seems like a bigger challenge here. That will probably become a large diff.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mthom/scryer-prolog/issues/615#issuecomment-816701523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFKHXWMX2FZ7ULKRAPQHKLTH4BU7ANCNFSM4OJ3LNXQ .

[tniessen]

There must be a better way than a huge diff.

If I figure out how to redefine the sdeq! macro and ~if this crate doesn't actually use any properties of SliceDeque that aren't provided by VecDeque~ (it does), it might be quite small. (I'm very new to Rust, so we'll see.)

Edit: the APIs of SliceDeque and VecDeque have subtle differences and are not interchangeable. make_contiguous can be used to access a VecDeque as a slice. I suppose the only way to solve this without causing an even bigger mess is to define another Deque implementation that either uses SliceDeque or VecDeque internally, but I'm struggling with the implementation.

[triska]

Is it worth trying to ask the maintainers of the affected crates for the implementation of the needed functionality to simplify this change? You can link to this issue to explain the motivation for the needed features.

[jacobfriedman]

Ah, the tangled web of dependency maintenance. Who's going to maintain the maintainers?

@triska, I remember in one of your PoP videos you spoke of depending on systems for not months, not years, but decades. I left the node/npm ecosystem because of careless package dependency.

How is it possible to lessen the dependencies so the installation is unaffected by external forces?

[tniessen]

Is it worth trying to ask the maintainers of the affected crates for the implementation of the needed functionality to simplify this change?

According to the maintainers, SliceDeque cannot work in WebAssembly because of the assumptions SliceDeque makes about memory management within the host system.

We can replace SliceDeque with VecDeque for cross-compilation to WebAssembly, but the APIs are not compatible. For example, the return type of ::remove is different, and, because VecDeque isn't backed by a slice, it doesn't deref into a slice, so the only way to access it as a slice is to call make_contiguous first.

It would be great if the APIs were compatible, but that seems impossible at this point, and would mean breaking changes for the maintainers of those crates.

So the best option might still be our own Deque implementation that only provides the features we need, and that either uses SliceDeque or VecDeque internally. That's the approach I went with so far, but I am struggling with a proper implementation. (Also, it might end up being super slow in WebAssembly.)

[mpabst]

@tniessen et al: Would you like some help finishing this? I'm interested in a good WASM Prolog too and have free time to contribute.

[triska]

@mpabst: I only want to add that I would greatly appreciate you looking into targeting WASM, and please let me know if I can help in any way by ensuring that the cryptographic libraries can be ported! It seems that simply outlining the steps and issues you encountered would already be a great contribution! Thank you a lot!

[triska]

@tniessen: Please have a look at the latest developments in the rebis-dev branch:

https://github.com/mthom/scryer-prolog/tree/rebis-dev

Notably, db20cda27aff6a13dc7f60b87549359b59b36f6e changes many occurrences of SliceDeque to VecDeque! Does that help for the WASM port?

[triska]

@tniessen: Please take a look at the latest commit, 68b4951bc97d3f6e7abc4d97b69dde519196b21e, which removes all remaining occurrences of SliceDeque! I hope it makes the WASM port a lot more feasible?

[triska]

Update: I have filed #1600 to eliminate the OpenSSL dependency. I hope this helps with the WASM port?

[triska]

As of #1907, the dependency on rug is gone. I hope this helps with the WASM port?