Support TypeScript/Flow type annotations

[dead-claudia]

Edit: Missed a difference...

This has come up before (#393), but I feel it could get resurrected again, since the syntax appears to be stabilizing. TypeScript would be much easier to use with macros, and Flow has kept a very TypeScript-like syntax as well. Babel supports Flow type annotations officially, and keeps them in its internal AST for plugins to manipulate. People have attempted to use SweetJS as-is with them, with varying degrees of success (generally little). And Closure Compiler also appears to be going in the direction of allowing TypeScript annotations as well. Could this use case be supported?

The type syntax overlaps tremendously between the two, and they only really differ in a few areas, mostly in semantics:

• TypeScript's {} vs Flow's mixed
• Their type-checking behavior is different with Object
• Flow's types are non-nullable by default, unlike TypeScript's.
• Flow tends to infer a lot more types than TypeScript.

As for pure syntax, you could parse them identically except for TypeScript's new () => T vs Flow's Class<T>.

[vendethiel]

Could this be fixed with a custom readtable?

[natefaubion]

Could this be fixed with a custom readtable?

No, because its basically impossible to tell from a read whether : is going to be for something like an object property, a label, or a type signature. This is something that has to be built into the expander so that it can recognize all these forms for purposes of hygiene. It also means macros that potentially match on binding forms (function args, vars, etc) have to know the type syntax otherwise they won't work.

[dead-claudia]

There are also other productions that would need to come in shortly thereafter, but this bug more or less blocks those.

• Type aliases
• Interfaces
• Generic classes

All three have the same syntax between Flow and TypeScript (mod the nit with constructors).

[disnet]

Yeah, I'm definitely open to this.

I think we would want to make sure our handling of the type language is generic enough to deal with both TS/Flow etc.

My main concern would be the added language complexity macro authors who want to handle functions/vars would need to deal with as @natefaubion pointed out.

[kkirby]

Any update on this?

[disnet]

Work is continuing on the redesign (#485). Once that lands it will make sense to think about adding type annotation support.

[elibarzilay]

+1, imported from Microsoft/TypeScript#4892

[KiaraGrouwstra]

disnet on gitter: "as long as your macro forms “look” like normal ES6 code you can pass it through TS/babel first just fine".

Trying out a ts -> babel -> sweet webpack workflow I still encountered a transpilation issue, but otherwise this option may be of interest for in as far we'd manage to cope with this limitation on external macro 'looks'.

[elibarzilay]

@disnet, Sounds like you're talking about macros that mostly look like function applications, which would be the most boring ones. (Ie, macros where you're playing with control flow, which are not too relevant now that the syntactic weight of wrapping stuff in thunks is tiny.)

[amir-arad]

We've released an alpha version of our TS visitors framework called tspoon . Tspoon should enable ts_parse-> sweet -> ts_transpile -> ... meaning you can run sweet as pre-transpile macros, and even disable specific transpiler warnings and the like.

[elibarzilay]

@amir-arad, IIUC, that project provides a hook into TS somewhere between parsing the input and producing the output. If that's correct, then it's kind of an option to implementing some macros on top of TS, so in theory it could be used to implement sweet.js for TS.

But I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation, which is why I think that the right way to go would be to extend sweet.js to recognize TS syntax and spit it back out (for TS to process).

[KiaraGrouwstra]

I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation

Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?

I think that the right way to go would be to extend sweet.js to recognize TS syntax and spit it back out (for TS to process).

Well, if one were to count outstanding ES proposals (e.g. decorators, type annotations, though I forgot which of my Babel plugins added that), I believe TS actually does not really add much syntax over what has been proposed for ES already -- the extent of 'not much', AFAIK, being decorators on parameters.

So in that sense, might those two roads not be closer to being one and the same?

[dead-claudia]

@tycho01

Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?

I know I wasn't the one you were asking, but I think Sparkler is a very good example of this. The TypeScript parser isn't particularly extensible, either (it's implemented in a giant closure).

Well, if one were to count outstanding ES proposals (e.g. decorators, type annotations, though I forgot which of my Babel plugins added that), I believe TS actually does not really add much syntax over what has been proposed for ES already -- the extent of 'not much', AFAIK, being decorators on parameters.

So in that sense, might those two roads not be closer to being one and the same?

You're correct in that TS doesn't add much syntax that isn't already either in or proposed for ES now. But type annotations are probably harder to parse than even the rest of the language.

[elibarzilay]

[@tycho01]

I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation

Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?

I'm talking about the technical level. The kind of things that a good macro expander needs from a representation of syntax can be very demanding on one hand, and subtle on the other. (And sweet.js is one of the very few rare cases of a good macro system.) I doubt that a generic exposure of syntax nodes would be as full as needed (but TBH I didn't look too deeply).

[@isiahmeadows]

The TypeScript parser isn't particularly extensible, either (it's implemented in a giant closure).

Right -- that when I figured that for the TS people to add macros would be wrong from all kinds of aspects. It requires certain skills that people who really care about syntax enough have (to know what hygiene means), but they probably don't. It's a whole layer of complexity that actually has very little with the goals of TS. And like Flow, it's coming from a neighborhood of people who don't care much about such things anyway.

And then I realized that this can be just like the Typed Racket case, where the typed language doesn't do anything interesting at the syntax level (besides typechecking, of course), and instead it just expands the code completely before it starts. So especially since the TS and Flow worlds have settled on a very similar syntax, this should really go as an extension of sweet.js which could then be consumed by one of these.

And in the typed racket case, there was some pressure from a few people to have some type information available to the expander, so it's possible to use that information to expand macros in different ways based on it. In that case there was certainly understanding of how useful this could be, but not enough human resources to do so (to intertwine typechecking with expanding macros). In the JS case, it would be interesting what happens but for now the "immediate" goal is to get people to notice sweet.js and stop the insanity of piles and piles of complete-source to complete-source "transpilers" and start doing the damn thing properly.

But type annotations are probably harder to parse than even the rest of the language.

It's probably not too hard for an environment that knows about identifiers etc. The real problems will happen in case of slightly different parsings being done by these things (TS vs Flow), but even in that case, sweet.js could just stick to some agreed lcd.

[disnet]

@elibarzilay knows what's up.

The right thing to do is extend Sweet with support for TS/flow syntax. This actually isn't too hard, certainly not nearly the same difficulty as the rest of the macro system. Just need to add a few cases in the parser and the codegen to handle type annotations.

The obvious concern with going down this path of supporting "downstream" languages in Sweet itself is that handling n languages in our parser (that really should just be macros) is untenable. I think TS/Flow are big and close enough to warrant this embrace and extend approach. Maybe eventually Sweet will get good enough for that other "e" :)

[elibarzilay]

I completely agree. I think that TS/Flow are exceptions since they are a language extension that is largely unrelated to macros. For other transformer libraries, it would be nice to start seeing them convert the code to go through Sweet as a much more logical choice as usual with macros being local transformation rules. When that becomes popular enough (and I might be idealistic here, but I'm convinced it will), then people will finally "discover"[*] that they can do some actual work as macros -- to the point of re-implementing TS/Flow on top of sweet.

([*] And this will definitely take some time. Even in the Scheme world, I remember talking to someone who was very surprised that in Racket we'd do something crazy like invoke GCC as part of the compilation -- in most people's minds, that's way beyond what a sane macro system should be able to do. They're just not used to separate compilation (Racket's phase separation) making things robust enough that anything goes.)

[KiaraGrouwstra]

Regarding TypeScript as "some babel modules + compilation errors" for a second, I'm getting the impression that, in a similar vein as you've mentioned, existing babel modules essentially represent ES-next syntax implementations that would be awesome to have available in Sweet as well.

This leads me to a question, answers likely clear to you guys, but not so much to me as an ignorant but curious user.

• if ES proposals / babel models represent new syntax desirable to incorporate into Sweet at one stage or another, do these existing babel modules contain the info/functions required by Sweet?
• If so, could they be used/converted to prevent duplication?
• Otherwise, what additional logic is required that existing babel module implementations do not expose?

If the additional information required would be trivial, perhaps it could become the norm for the community to provide implementations supporting both babel and sweet for future ES proposals.

[dead-claudia]

I was thinking: would it be a better idea to, instead of adding more and more things to the parser, allow an Acorn-style syntax extension mechanism? That might work better, since you can validate anything you come across. Maybe something like this?

// Just validates.
syntaxdecl async = function (ctx) {
  // ctx.expectCallable(true or undefined) -> include newlines
  return ctx.expectCallable(true) && ctx.is("expr")
}

syntaxdecl inline `::` = function (ctx) {
  return ctx.next("expr") &&
    ctx.next(this) &&
    ctx.next("expr") &&
    ctx.is("expr")
}

[disnet]

Right, so worth taking a moment to explain the philosophy of Sweet (and macro systems in general).

Consider the following languages:

• ESyyyy, where yyyy is the current year, is defined by the standard ECMA262
• ts, can mostly be thought of as ES2015+types
• flow, can mostly be thought of as ES2015+types
• babel, can mostly be thought of as ESyyyy+extras where extras are things like jsx, object spread, and various other pre-stage 4 proposals (at which point we get ESyyyy+1) and experiments

Sweet is a parser for yet another language: ESyyyy+macros. The cool thing about the +macros part is that you can define syntax transformers (aka macros) which allow you to accept many more languages than just the strict grammar defined by ESyyyy+macros would imply.

The goal of Sweet is to support the creation of all the languages listed above (and many more) via syntax transformations that you can compose and reason about (acorn’s plugin approach is a non-starter because it is neither composable nor reasonable).

A non-goal of Sweet is to bake in support for every JS-adjacent language into the parser. That’s what macros are for.

Practically, the kinds of syntax transformations Sweet currently supports are not powerful enough to implement all the languages listed above. That said, the plan is to make them powerful enough. Operators (**, :: etc.) can be handled with #517, more complex operator-like forms can be handled with the re-introduction of infix macros, jsx can be handled with readtables, types can be handled with modules #233 + some stuff from Racket’s syntax zoo (the syntactic form of types is straightforward to handle it’s the type checking that requires the extra stuff), object spread can probably be handled by something like Racket’s implicit forms.

Basically, the roadmap is steal everything from Racket.

What we’ve been discussing in this thread is extending Sweet to support the language ESyyyy+macros+types as a temporary kludge because the +macros are not (yet) powerful enough to implement the +types. The only reason we’re considering this is because TypeScript/Flow is a relatively big enough deal for people who might be interested in using macros. It’s a way to bootstrap us into taking over the world (muahahaha).

[dead-claudia]

@disnet Okay. I'm not that invested in that idea of extensible syntax, anyways.

It’s a way to bootstrap us into taking over the world (muahahaha).

:laughing:

[KiaraGrouwstra]

To expand on my earlier comment, if all ES-next proposals would require non-trivial reimplementations as macros, I suppose that means Sweet would essentially be competing with Babel for features added. If so, then adopting Sweet unfortunately for the time being becomes more of a trade-off rather than a straight-forward decision. Obviously, its potential is orders of magnitude bigger, so I definitely hope Sweet could overtake Babel as the go-to way of implementing new features. Until that time, there may be a threshold of momentum though. But then again, I guess for now the bigger step is still that Racket roadmap...

[dead-claudia]

@tycho01 Almost every ES proposal at the syntax level is possible to implement, and many already have with previous versions of Sweet. This includes ES6 classes in their entirety and arrow functions.

Oh, and Babel isn't likely to disappear. Compilers are much better at semantic optimization and even implementation correctness than macro processors in terms of specifications. Good luck implementing generators in pure Sweet. :wink:

[vegansk]

Are there any progress with typescript support?

[disnet]

@vegansk various background tasks have been completed but nothing directly on supporting TS/Flow in sweet.

I'm definitely motivated since everything I write now is in flow (even sweet core!).

My current focus is updating our internal AST to the latest version of Shift (so we can support async/await). Once that has been handled we will be in a good position to support types.

If anyone wants to help out a good place to start is getting TS/Flow support added to Shift. We depend on shift codegen to render our AST so it will need to be extended to handle types.

[gabejohnson]

@disnet are you thinking just matching what Babylon does (maybe more granular TypeAnnotation node types)?

[disnet]

@gabejohnson I haven't looked into it in any depth yet but yeah that would probably be my initial approach.

[slotik]

@disnet I'd like to help with this. I looked at the Swift issue list but couldn't see any feature request related to type annotations. Should I create such a request or is there a better way of adding the support you need?

[disnet]

@slotik no need to open a shift issue. I chatted with them (sorry, forgot to update this thread) and the shift project is uninterested in type annotations in their core AST and that's actually fine for our purposes; we are already extending the shift-ast-spec in a couple of places (syntax declarations and import for syntax) so a few more is not a problem. The only code from shift we are really depending on is shift-codegen so once we get to the point of implementing type annotation support we will probably need to fork it.

As far as what we can do right now I'm sort of blocking anyone helping out with code. I'm in the middle of refactoring sweet-spec and sweet-spec-macro, which will have far reaching consequences in sweet-core.

What would be super helpful though is taking a look at the shift-spec and proposing what nodes we should add/change to support annotations.

[slotik]

@disnet I started to look at shift-spec and I made a couple of notes around the places that seem to need changes:

https://github.com/slotik/shift-spec/commit/980114c03a1f956abd3e0190d89105cd09807ee3

If this looks like it could provide some value, I can try to suggest concrete modifications as well as a new set of nodes to express interface and type statements.

[KiaraGrouwstra]

What would be super helpful though is taking a look at the shift-spec and proposing what nodes we should add/change to support annotations.

For reference, TS AST nodes.