nodejs
How could we support typescript without vendoring it?
I would like Node.js to support the following user experience
$ node script.ts
Typescript support is missing, install it with:
npm i --location=global typescript-node-core
$ npm i --location=global typescript-node-core
...
$ node script.ts
"Hello World"
(I picked the typescript-node-core name at random, I would be extremely happy if that could be ts-node)
Note that script.ts could evaluate to either cjs or mjs depending on the tsconfig.json file.
Originally posted by @mcollina in https://github.com/nodejs/node/issues/43408#issuecomment-1182883689
Why vendoring is not an option?
- TypeScript changes too fast and they do not follow semantic versioning, nor do they provide any long-term support.
- We will never be able to pick a default
tsconfiggiven the 100+ options.
So basically you would like Node.js to automatically load something from a global predefined place when it starts?
So basically you would like Node.js to automatically load something from a global predefined place when it starts?
Yes, but only when started with a .ts file.
Wasn't that the main purpose of custom loaders? You should even be able to set that in a NODE_OPTIONS.
Please don't start issues related to loaders without tagging @nodejs/loaders
This is not related to loaders. This is about the developer experience of Node.js. Loaders are custom, user-specified components that are started before the app. I'm talking about shipping something that would provide a better user experience for TypeScript users without additional configuration.
Please keep this issue about the user experience and not specific implementations.
I'm trying to see how this isn't reinventing the wheel, but I'm not getting it. Could you please explain how it's substantially different?
How I'm seeing it is: This is already easily achieved via Loaders with next to no effort (and that effort is basically set-and-forget). We have a simple working example of it in node/loaders-tests/typescript-loader. For a more robust one, ts-node provides such a loader (ts-node/esm). And what we currently have via loaders is super fast. We just switched to it at my work and saw like an 800% speed improvement.
PS In case we veer into the territory: I would vehemently object to TypeScript support in Node itself.
I'm trying to see how this isn't reinventing the wheel, but I'm not getting it. Could you please explain how it's substantially different?
I would rather stay away from discussing a specific implementation of this. This could be loaders but it doesn't matter. I care about listening to our users. Once we agree on the user experience, then we figure out what's the best way to ship it. If it's already possible as you hinted, it would be terrific.
PS In case we veer into the territory: I would vehemently object to TypeScript support in Node itself.
That's what our users are asking for. We cannot provide TS directly into Node.js core for plenty of reasons (it's just not possible for the tsconfig mayhem), including the fact that the TS team think it's a bad idea. I propose that we implement the user experience described in https://github.com/nodejs/node/issues/43818#issue-1303418894.
A few questions come to mind:
- Are we making an exception for TS only, or are we open to adding more file extensions custom error message in the future?
- How do we chose what tool to recommend?
- Is Node.js core the correct place to solve that kind of problem?
Currently, what happens when someone runs node script.ts depends on the env:
- if we're inside a
{ "type": "module" }package, it will throw as.tsis not a recognized extension – in this case, it's quite easy to add a custom error instead as you're suggesting; - otherwise, it will parse
script.tsas a CJS module – in this case, we could try to parse the file as a CJS module, and if that fails display the error message you're suggesting?
We would then need a way to detect that typescript-node-core program is installed on the local env, and to defer to it to load that file. Maybe by making a $PATH lookup?
In an app with "type": "module" in its package.json, you get this:
node script.ts
node:internal/errors:477
ErrorCaptureStackTrace(err);
^
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /private/tmp/test/script.ts
at new NodeError (node:internal/errors:388:5)
at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:80:11)
at defaultGetFormat (node:internal/modules/esm/get_format:122:38)
at defaultLoad (node:internal/modules/esm/load:21:20)
at ESMLoader.load (node:internal/modules/esm/loader:431:26)
at ESMLoader.moduleProvider (node:internal/modules/esm/loader:350:22)
at new ModuleJob (node:internal/modules/esm/module_job:66:26)
at #createModuleJob (node:internal/modules/esm/loader:369:17)
at ESMLoader.getModuleJob (node:internal/modules/esm/loader:328:34)
at async Promise.all (index 0) {
code: 'ERR_UNKNOWN_FILE_EXTENSION'
}
So to provide the UX you’re describing, we would have to add a special case within this error flow where if the unknown extension is .ts, we print a special message. This will inevitably raise the question of what other unknown extensions do we want to print guides for; or we could print some message for all unknown extensions along the lines of “go to https://nodejs.org/guide-to-loading-non-javascript-file-types” and that documentation page could be a clearinghouse of instructions for various types.
In an app without "type": "module", however, script.ts is parsed and executed as CommonJS. If it happens to be runnable JavaScript, it runs. Otherwise it’ll error when V8 tries to parse TypeScript syntax. So to provide a similar experience in CommonJS, within that error path you’d have to add a check that the file that couldn’t be parsed had an unknown extension, and then print the message. Keep in mind that there’s a type annotations proposal that would make many TypeScript files into runnable JavaScript.
Either way, the solution that we would be recommending to users would involve adding the TypeScript loader. And transpiling TypeScript is squarely one of the use cases for loaders. So I object to treating this as “not related to loaders” and removing that label.
@GeoffreyBooth let's imagine that the solution ends up looking like that (oversimplification):
if (entryPoint.endsWith('.ts')) {
process.argv.unshift(entryPoint);
entryPoint = '~/.node/typescript-node-core/bin.js';
}
Sure typescript-node-core will probably use loaders, but that's arguably an implementation detail. (FWIW I agree that a clean solution should involve loaders; if anyone is interested to work on this, that's where I would suggest them to explore)
What if it were some kind of…plugin (idk what the official term is relative to Node.js), like crypto or intl, with which node could be compiled or not? We could then have however many and we aren't responsible for them. It would be very similar to loaders, but a bit different and i think perhaps a little closer to what Matteo is talking about.
Also, this would avoid incorporating typescript-specifics (eg their .ts vs .js file extension nonsense) into node core (which is my main concern).
Annnnd it wouldn't require the CLI args everyone laments about loaders.
Yeah, I think the important factor is user experience. Not sure how loaders work exactly but a DX like this would work - it could be specified in a configuration file which loader to use given an extension:
.m?ts --> use typescript loader
.cf --> use coffeescript loader
.... ---> any other user defined type
This could have global defaults and local (package.json) overrides. The global defaults would be shipped with node.js.
Maybe cli flags could be passed to the loaders, when needed. This way we can load any file conveniently with node path/to/file and spawned processes will also do so without additional config.
One thing to keep in mind:
I see in comments this is described as triggered by the entrypoint's file extension. There are situations where you may be running a .js entrypoint but still want a TS-enabled runtime, for example my-cli which executes my-cli.config.ts.
...but that's a detail.
High-level ask from users is still: "How do I opt-in to my runtime having automatic TS support?"
When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for <insert non-TS thing here>?
I am in favor of this and I think we are not doing a good enough job. I like Matteo's proposed idea of a blessed "how to run TypeScript module" - ideally I'd also like us to provide a node build or similar on top of it so people have a workflow for CI.
We'd probably need:
- To bikeshed/figure out the way to run TypeScript without extra flags in a blessed way (a hook and pre-installed package sounds reasonable)
- To figure out the story for passing flags (like transpileOnly) to TypeScript
- To figure out the production story (e.g. node build myProject) that will produce a .js file
Might also be interesting to know if this unlocks optimizations?
When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for <insert non-TS thing here>?
The sense in general as well as the trend from every other runtime is: Users are asking for better TypeScript experience a lot and almost no users asking for other type systems (like flow) or languages (like reason). Heck, someone opened an issue on this today
When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for <insert non-TS thing here>?
Not at this point. I would be happy to evaluate other languages (like coffeescript), but I do not think any other language is as popular as TypeScript. Anyway, I think our implementation should allow for more extensions to be considered for inclusion.
More radical: bundle swc and just run the code. :-)
(Caveat emptor: swc is a transpiler, not a type checker. But on the flip side, it also handles JSX and TSX.)
More radical: bundle swc and just run the code. :-)
(Caveat emptor: swc is a transpiler, not a type checker. But on the flip side, it also handles JSX and TSX.)
That's not so crazy to me. It's sort of what I was suggesting above: https://github.com/nodejs/node/issues/43818#issuecomment-1183579118
After seeing how other emerging runtimes provide native TypeScript transpilation by default, I am in favor of @mcollina's proposed idea. I'd like to see Node support running TypeScript code, ideally out of the box, with no additional flags or options.
More radical: bundle swc and just run the code. :-)
(Caveat emptor: swc is a transpiler, not a type checker. But on the flip side, it also handles JSX and TSX.)
Does swc have a LTS policy compatible with Node.js one? If not, I don't think vendoring is an option.
@bnoordhuis
More radical: bundle swc and just run the code. :-)
Maybe sucrase? It's not getting as much traction but it's a pure JS tool and it's faster.
I also have positive experience with SWC
When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for <insert non-TS thing here>?
yes, I did a community poll and 65% replied they use TypeScript in node.js. 35% picked JavaScript.
I appreciate that we're admitting it's by far requests for TypeScript and almost nothing else. Sometimes toys like coffeescript transpilers are used as justification that node's current APIs (loaders, etc) are sufficient. My experience is, they are not, and the challenges only arise when truly attempting to support typescript for a large number of projects. The focus on TS is healthy, because it will make node's APIs better. (and that includes being better for coffeescript and other stuff) EDIT the preceding paragraph is not great. I attempted to articulate my thoughts more clearly down here: https://github.com/nodejs/node/issues/43818#issuecomment-1184686030
In a few threads that ts-node's user experience is cited in favor of node's current APIs, --loader or otherwise. But we jump through a bunch of hoops to make that work, and node often makes it harder. Node's current APIs are not good enough.
Coming back to the idea of a blessed "how to run TypeScript module," I think we should make a list of all the hooks this module will require to do its job. One way to start: we can list all the hacks that ts-node goes through, all the places where we duplicate node's logic because node doesn't expose a proper API.
Here are a few off the top of my head:
- repl
- tab completion (current api insufficient)
- top-level await (mechanism not exposed)
- inherit configuration blob to worker_threads and child_processes (currently requires hacks)
- stack trace support (fix bugs & perf in node's current implementation)
- stack trace cache setter (haven't looked at this one in a while)
- proper APIs into node's CJS exports discovery mechanism (discover ESM named exports in CJS source code, no way for transpiled code to contribute to this process)
The list is incomplete but hopefully illustrates the problem from a hook author's perspective.
Maybe sucrase? It's not getting as much traction but it's a pure JS tool and it's faster. I also have positive experience with SWC
SWC and sucrase are both transpilers. To get the most out of TypeScript, they will have to be paired with a type checker.
It sounds to me like ts-node has already done most of the work. Couldn't it be combined with node to provide a seamless user experience?
Note that ts-node also supports the transpile-only swc/sucrase use-case. And avoiding typechecking does not avoid the need for the missing API surfaces described above.
Just want to clarify those things to avoid any potential confusion.
I appreciate that we're admitting it's by far requests for TypeScript and almost nothing else. Sometimes toys like coffeescript transpilers are used as justification that node's current APIs (loaders, etc) are sufficient.
Note, I personally value our users using other languages on top of the platform (be it flow, coffeescript, reason or anything else). I think it's important that a solution here would not hurt the development experience of those users. I also don't agree with the charactarization of CofeeScript as a "toy" (any more than Node is).
That said - absolutely: the feedback we've been getting from the community has overwhelmingly been in favor of TypeScript.
I also see a strong case for transpilation + swc over tsc and type checking by default since that's what the current proposal suggests.
Fair enough, and someone called me out on the same in a separate chat. I think I can clarify what I was thinking:
My belief is not that coffeescript itself is a toy, and "toy" was poor word choice to begin with.
I think there's a risk when discussing node hooking APIs to only look at simple use-cases and stop there. This can give the mistaken impression that an API is complete because it meets a simple use-case but not a more complex one. A coffeescript loader hook is an example of one that's simple to implement and does not have some of the complexity that exists in TypeScript hooks.
I think a focus on robust TS hooks will be healthy for node, healthy for fans of TS, coffeescript, flow, etc. Because it leads us to design feature-complete APIs that can handle demanding use-cases.
I'll put a little EDIT on my previous comment linking to this one, I hope this makes sense.