Release 0.8?

[Yoric]

I'm currently working as a background task on a bridge between BinAST and Ratel. However, Ratel 0.7.0 and Ratel 0.8.0 seem to be very different beasts, with very different ASTs.

Which version should I target? Is Ratel 0.8.0 meant to be released soon?

[maciejhirsz]

Definitely target 0.8! We should at least split out the AST, Lexer and Parser out so those can be pushed forward without the whole thing being stalled due to the transformer lagging behind.

[maciejhirsz]

I was thinking about this after replying here. Maybe it would be a worthy goal to set up a single, semi-standard, AST structure for all of the Rust projects? The one present now in Ratel I'm happy with, but it's not extensible (for things like TypeScript types or JSX) without changes, and it is bound to Arena allocator from toolshed, which makes it really really fast to generate, but also not as easy to manipulate, and it might not even be very feasible in all scenarios (such as language servers).

[Yoric]

That sounds like a good idea.

I can easily extract one from the spec of BinAST we're submitting to TC-39. Although the specs might change a bit between now and (hopefully) acceptance, they have been pretty stable for the past ~6 months.

edit It's not entirely clear how TypeScript or JSX would interact with the AST. Are they new nodes? Should it be possible to put a TypeScript node as, say, a Statement or an Expression in an ES6 AST?

[maciejhirsz]

JSX would be a new Expression variant, TypeScript is a bit more invasive as it requires both new nodes (type/interface declarations, namespaces etc.) and expansion on existing structures (appending types to function paremeters etc.).

I'm thinking if all of that can be made generic. I'm also reading through the de part of serde to see if we could apply the same model to the parser itself, so that it doesn't actually produce any AST at all, but emits events that can be used to produce an AST. I think such approach should also make it possible to create a source code -> BinAST conversion without going through the AST construction phase.

[Yoric]

JSX would be a new Expression variant, TypeScript is a bit more invasive as it requires both new nodes (type/interface declarations, namespaces etc.) and expansion on existing structures (appending types to function paremeters etc.).

Well, I guess it could be new function declarations/expressions.

I'm thinking if all of that can be made generic.

I think it can. For BinAST, we're generating the AST from a webidl. We "just" need a webidl for es6, one for JSX, etc. and the spec that if the same sum type name (e.g. Expression) appears in several webidls, then we need to generate a single sum type with all the branches that appear in all webidls.

[Yoric]

I think such approach should also make it possible to create a source code -> BinAST conversion without going through the AST construction phase.

That would be very nice. Also the opposite: a Ratel API that would let us output JS code, ideally without having to (visibly) go through an AST construction phase.

[Yoric]

Also, filed https://github.com/binast/binjs-ref/issues/235.

[maciejhirsz]

Ok, so I played a bit in the playground and basically copied the deserialization from serde:

https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=79f0b07ceb8d797583829895e1b8f46c

Couple observations:

1. This is already quite hard to follow even for this toy AST I've made.
2. Bolting error handling on this might be painful (unlike serde, we want to continue parsing even through errors, accumulate those and send them in a batch for better UX).
3. This only works for returning values. Technically you can build a stateful Visitor that can write to internal buffer and always return (), but that sounds quite hackish.

More research is needed I suppose :)