Research using Go binaries

[rglover]

There's a lot of potential for using Rust to scale compute-heavy work in Joystick. What I'd like to figure out is if we can wire up a simple abstraction for writing Rust code inside of a Joystick app and have it properly compile for target OS's (e.g., in your settings..json file, you specify which platforms you want to build for—MacOS, Windows, Linux, etc). Then, make it so you can just do:

import { rust } from '@joystick.js/node';

const calculate_revenue = rust.load('<name of binary>');
const revenue_data = await process.databases.mongodb.collection('revenue').find().toArray();

calculate_revenue(revenue_data);

Early experiments make me think this isn't just possible but relatively easy once you get past the build gotchas (I ran into issues with building Rust for Apple Silicon which were just a matter of ensuring the correct flags were passed at build time).

The advantage here is making Joystick scalable beyond the current limits of Node.js.

[rglover]

The more I dig into this, the more I think Go will be the better initial choice (easier syntax to pick up for JS devs, comparable performance, easy compilation to different platform targets).

Just speculation for now but initial read on it makes me think it'd be better to start with that and then add Rust and others laters as neccessary.

[rglover]

Backpeddling on this. After a bit more research, Rust is preferred for what I'm after (a way to offload heavy work from JS). It's a bit more flexible in terms of what you can build which is best for this sort of feature.

API wise I'm spitballing:

const result = ffi.rust('binary_name', {
  input: { ... },
  events: { on_success, on_error, on_warning }
});

To the on_error callback, we pass the error and the ffi invocation/instance. On that instance, we can do something like ffi.retry() so that way a process doesn't die and hang.

Alternatively, we can do something similar to BEAM where we create a "supervisor" layer that lets you run Rust code in the background at startup time:

const app = joystick.app({
  ffi: {
    background: [{ binary: 'binary_name', input: {}, events: {...} }],
  },
});

The idea being that Joystick will sidecar anything in the background array and make it accessible globally in the app via process.ffi.<language>.<binary_name>. The point being that once loaded/running, assuming it's a long-running process internally I can use that pointer to interact with/send messages between your Rust module.

A LOT of details to work out on this but the idea is making sense and taking shape. Best part is once this works: hot damn will Joystick rip through requests.