Browser/Node.js packaging story

[nornagon]

I've been thinking about the best way to go about packaging automerge-rs such that it can become the One True Automerge Backend, used by default in mainline automerge, and installable with no more fuss than an npm install. Here's what I've learned so far, and what I think the path forward could be.

The majority of the groundwork for this is already laid by the automerge-backend-wasm crate, using wasm-pack. The tricky part is getting the wasm bundle that wasm-pack generates to function seamlessly in the various JS environments that automerge currently supports (i.e. Node.js and web browsers via a bundler such as webpack). The tooling in this area seems to be relatively immature (see e.g. https://github.com/rustwasm/wasm-bindgen/issues/2265), so sadly this is cutting-edge rust+wasm+js technology. I expect this story to improve significantly in the coming years.

wasm-pack is able to generate output for a variety of different targets, most importantly, "bundler", "nodejs" and "web". Notably, none of these output formats work for both Node.js and the browser. See https://github.com/rustwasm/wasm-pack/pull/705 for a long discussion and a variety of workarounds. In particular, I think this workaround is probably the best way forward for automerge-rs: https://github.com/rustwasm/wasm-pack/pull/705#issuecomment-529705607

e.g.

package.json:

{
  "name": "automerge-backend-wasm",
  "files": [
    "pkg-bundler/index_bg.wasm",
    "pkg-bundler/index.js",
    "pkg-bundler/index.d.ts",
    "pkg-node/index_bg.wasm",
    "pkg-node/index.js",
    "pkg-node/index_bg.js",
    "pkg-node/index.d.ts"
  ],
  "main": "pkg-node/index.js",
  "module": "pkg-bundler/index.js",
  "types": "pkg-bundler/index.d.ts",
  "sideEffects": "false"
}

and then building with:

wasm-pack build --target bundler --out-dir pkg-bundler
wasm-pack build --target nodejs --out-dir pkg-node

The downside of this approach is that it builds two distinct wasm bundles. The bundles are nearly identical but differ in a few minor details. This is a fairly minor downside I think, as in a production app, in Node.js install time is not a big concern, and in a web app, the bundler will only pull the web version of the wasm bundle to the client.

With this in place, the following should work:

import wasmBackend from 'automerge-backend-wasm'
import Automerge from 'automerge'

wasmBackend.initCodecFunctions(Automerge)
Automerge.setDefaultBackend(wasmBackend)

// ... use Automerge, but it's fast now

With node.js it works out of the box, but with webpack you need to enable the experimental asyncWebAssembly feature.

Ideally all this would be under the hood in automerge and simply importing stock automerge would get you wasm-backed goodness. Unfortunately, the ecosystem is just not there yet. However, I think we can still support automerge-rs as the primary backend for automerge via asm.js. The setup we'd need to accomplish this is a little arcane in order to avoid needing complicated "tree-shaking" / dead-code-elimination strategies to ship the right code to clients, but I think during the transitional period, the package structure of automerge could look like this:

automerge
└── automerge-backend-asmjs

automerge-wasm
└── automerge-backend-wasm

i.e. two automerge main packages, the "default" one backed by automerge-rs compiled to asmjs, and a separate one backed by the wasm version, for those who want to make the tradeoff of dealing with wasm bundling issues in exchange for performance improvements.

Another possibility could be to arrange the packages like this:

automerge
├── automerge-backend-asmjs
└── automerge-backend-wasm

and have require('automerge') return an asmjs-backed API. To pull in the wasm version, you could require('automerge/wasm').

The API I'd really prefer here is something like the following:

const Automerge = require('automerge')
Automerge.setBackend(require('automerge-backend-wasm'))

However, I don't see a path to having a "pluggable" backend that doesn't do one of the following undesirable things:

1. Increase boilerplate for all users of automerge (i.e. require "registering" a backend)
2. Always ship the default backend to clients, even if you never use it

Hence the various package arrangement strategies above.

[epompeii]

This seems like a great start towards the goal of having a single backend! As to your first undesirable trait though, could we create a more intelligent default backend based on the environment and allow for overrides? It seems like a one line check is all that's needed to add these smarts for browser vs node: https://github.com/iliakan/detect-node/blob/master/index.js#L2 This would then shift things from being boilerplate to configurable. It may be useful for #34.

When looking into this, I saw that removing asm.js support has at least been raised: https://internals.rust-lang.org/t/should-we-downgrade-drop-the-asmjs-target/9835 I think we both agree that asm.js is a stopgap while the wasm ecosystem matures (and time works its magic on any lingering IE support requirements), so this probably isn't worth worry about today, just keeping an eye on.

[nornagon]

could we create a more intelligent default backend based on the environment

the problem is, you have to be able to convince the bundler not to include the default backend when it's overridden. so it has to be a static compile-time check. if we can find a way to do that then i'm all for it.

good to know that asmjs is on its way out. hopefully it doesn't disappear before wasm support gets good.

[alexjg]

This is a great write up, I love it. I think the majority of Automerge clients right now are either using a bundler with tree shaking abilities, or they are applications which are not loading their code over the network anyway (i.e Node or Electron), so I don't think having both the WASM bundles in the package is a problem.

Vis a vis doing all this under the hood of Automerge: I completely agree that this is the ideal situation and we want to figure out howto get there. The main reason we can't do that now - and I think the reason you hint at in your ecosystem comment - is that chrome doesn't allow us to import WASM modules larger than 4kb in the main thread. asm.js does allow us to get around that but I vaguely recall that the asm.js file size was much larger, although I will have to check that. Effectively I think this:

const Automerge = require('automerge')
Automerge.setBackend(require('automerge-backend-wasm'))

has to live in a service worker.

Anyway, I'll follow up with some feedback on the PR.

[orionz]

Sorry to be so late to the conversation here. I'd like to reopen this conversation as I need to resolve the automerge-wasm packaging story shortly.

I don't agree that we should use asm.js. In my limited testing of it - it's been larger and slower than the native js implementation and therefore is never really going to be a valid candidate to replace it. It wont be adopted if its functionally worse and there's no point in creating it if no one will adopt it.

I don't see any option but to embrace the async nature of wasm. We could make an automerge-backend-wasm package that does an async require and can be passed into setBackend and/or create an automerge-wasm which also has an async require and does exactly that for you. There is also the fact that Node 14 has top level await which opens up some other options as well.

I'm no javascript packaging guru so happy to hear more ideas.

[nornagon]

@orionz I'd be happy to see a solution that doesn't require asm.js, but I think the dichotomy in the original post still stands:

However, I don't see a path to having a "pluggable" backend that doesn't do one of the following undesirable things:

1. Increase boilerplate for all users of automerge (i.e. require "registering" a backend)
2. Always ship the default backend to clients, even if you never use it

[orionz]

I just published automerge-wasm npm package which works in both node and web apps. The automerge-js package featuring the classic API will be coming soon

[fo-fo]

Has any thought been given to React Native? I think that's one definite downside of the automerge-wasm approach compared to the old JS-only implementation of Automerge because React Native doesn't have well defined WebAssembly support yet. I guess the optimal approach would be to compile the Rust code to a native (non-WASM) library for React Native, with non-WASM bindings to the JS land, but that's non-trivial to implement.

Just posting this as a reminder that web/Node.js are not necessarily the only platforms that the users (including me) want to support.

[alexjg]

My current plan for this is to first build an Android and Swift wrapper (we need this anyway) and then to build a React Native bridge which is based on top of those wrappers. No timelines on that yet but that's the general idea.