Refactor as a Node module

[Yoric]

We should use Neon to expose this library to Neon. This would open the library to a larger community and save us from all JSON (de)serialization and IPC communication.

[xtuc]

Another alternative would be to use wasm-bindgen and compile it to WebAssembly.

I would also be interested in adding a binast output in Babel. We just need a way to call it from JS.

[sendilkumarn]

@xtuc I had this idea, But I am afraid with the amount of JSON works that we are doing here, it would result in a huge binding file with wasm-bindgen (when we need some boundary crossing).

With Neon, the boundary crossing has a performance impact not a memory impact, which will force us to reduce the boundary crossing. WDYT?

[Yoric]

Regardless, we'd need to refactor away the parsing. Parsing would look very different in a JS-based world.

[Yoric]

@xtuc, @sendilkumarn does either of you know if there is a good way to send a JSON object to the webassembly world? Or does this need to wait for TypedObjects (or whatever they are called now)?

[xtuc]

Good question, i'm not aware of any simple solution.

I would put the JSON string into the memory and extract/parse it from JS.

[Yoric]

So, the steps would be:

1. parse with Node-level parser (Babel, Shift, ...);
2. convert the parsed object into BinAST's ES6 format;
3. JSON.stringify the result;
4. call the wasm module;
5. parse the string from Rust.

Is that it?

edit Removed word "native".

[xtuc]

Ah I had the other way around in mind, sorry.

1. Native is super confusing, you mean from existing tooling? Like Babel
2. and 3. will be very expensive?

Before the 4. you need to put the JSON string into the module's memory (using the WebAssembly.Memory object in js).

I'm sure that we can find a common more efficient format between JS and Rust, similar to https://github.com/firejune/struct.js/tree/master for C.

[Yoric]

Native is super confusing, you mean from existing tooling? Like Babel

Edited to remove the word "native".

2. and 3. will be very expensive?

2. is one tree walk, should be reasonable;

3. is pretty expensive, although given the number of things we do in the Rust side, I'm not sure it's the critical cost.

[xtuc]

Yes I agree, it's at build/compile-time so the performance shouldn't be critical.

[sendilkumarn]

2. convert the parsed object into BinAST's ES6 format;

Do we have some parser already available for this?

[Yoric]

Do we have some parser already available for this?

That's the role of shift::FromShift. It's imperfect and currently limited to a subset of ES6, but it works well enough for the moment.

[sendilkumarn]

IIUC

1,3,4 >> Happens in JS 2,5 >> Happens in Rust

Is that correct ?

[Yoric]

Yeah, but 2 should probably move to JS.

[xtuc]

That's a good question, I would assume that 2 would be faster in Rust and doesn't has to be in JS.

[sendilkumarn]

if 2 is faster in rust, then won't it be better to handle 2,3,4,5 in Rust. I am worried about the boundary crossing...

Else we have to rewrite 2 in JS

[Yoric]

If we want the core to work with several parsers, 2. should be kept close to the parser, so in JS.