ts-node with project references in a lerna monorepo

[kerdany]

I'm using lerna with typescript project references for a node application containing two packages. Package lib and package app (which depends on lib).

I've configured typescript project references so that package lib is built automatically whenever we run tsc --build inside package app (i.e. to build for production).

Here's how the tsconfig.json files are configured for each of the packages:

packages/lib/tsconfig.json:

{
  "compilerOptions": {
    "strict": true,
    "rootDir": "src",
    "outDir": "dist",
    "tsBuildInfoFile": "dist/.tsbuildinfo",
    "composite": true
  }
}

packages/app/tsconfig.json:

{
  "compilerOptions": {
    "strict": true,
    "rootDir": "src",
    "outDir": "dist",
    "tsBuildInfoFile": "dist/.tsbuildinfo",
    "incremental": true
  },
  "references": [
    { "path": "../lib" }
  ]
}

Currently, running tsc --build (inside app) compiles typescript to javascript in the dist directory in each of app and lib perfectly fine, the setup runs flawlessly in production mode.

The problem, however, is that trying to run the project in development with ts-node via nodemon --exec 'ts-node src/index.ts' in development fires the following error:

/path/to/packages/app/node_modules/ts-node/src/index.ts:245
    return new TSError(diagnosticText, diagnosticCodes)
           ^
TSError: ⨯ Unable to compile TypeScript:
src/index.ts(1,23): error TS2307: Cannot find module '@myproj/lib'.
...

What seems to be happening, is that ts-node is looking for the @myproj/lib package inside node_modules directory (symlinked by lerna), instead of compiling it on the fly through typetcript's project references, as is setup inside both tsconfig.json files.

I validated my theory by:

• Running a regular tsc --build first (which also builds the lib/dist code).
• Then running nodemon --exec 'ts-node src/index.ts' again, and it ran fine then.

Which means that ts-node in this case is loading lib via the compiled .js code inside lib/dist (symlinked by lerna), NOT via compiling its .ts code on the fly (via references).

I'm using [email protected] (currently the latest version).

Some questions:

• Doesn't ts-node currently support project-references yet or passing the --build flag to tsc yet?

• Am I doing something wrong in my (e.g. tsconfig.json) configurations that's causing ts-node not to compile/build lib.

• I can see that a new --build flag is being added to ts-node in the master branch (commit), but it seems that it's irrelevant to tsc's --build flag.

Note: Currently I'm working around this (without ts-node) via nodemon --exec 'tsc --build && node dist/index.js' till I get this figured out.

Thanks!

[blakeembrey]

1. I don't think so, though I can replicate project references working for other cases. It's using the language services API today which has typically had no problems, but I'd have to ask if you can build a simple repro for me to work with on this to confirm.
2. I don't think so, more likely that references don't work as expected.
3. Correct, I can probably rename to --emit to clarify the use-case. However, I run into issues with project references on the latest master build that need to be fixed (so your example might help me too) 😦

[kerdany]

Thanks @blakeembrey

1. Sure, I will prepare a sample repo to reproduce the issue first thing in the morning (UTC+2 here).

Thanks!

[kerdany]

Hi again, I've created code to reproduce issue here: https://github.com/kerdany/ts-node_issue-897

[blakeembrey]

@kerdany Thanks! Interestingly I get the same error in development with VS Code and tsc, until I enable --build. This makes sense, since I assume that TypeScript is actually building references first when you specify --build. It does make it trickier for ts-node though, I guess it should also build those dependencies somehow. We might have to do something in memory to make this possible...

[blakeembrey]

Another interesting part of this is that it's related to node and TypeScript's understanding of module resolution. You can actually make this work with the current release of ts-node by importing directly the other .ts file:

import { greet } from "@myproj/lib/src/index";

Edit: Don't expect this to work in the next major version though, I've been having a lot of trouble with getting project references working and this also fails with VS Code tsc.

[jomi-se]

Have you done any progress or got any clues for this? I just ran into the same issue as well

[impaler]

I also ran into this trying out packages in yarn workspaces. When wanting to debug with node -r ts-node/register as a work around for now I am composing a root tsconfig.json with all relevant references and running a tsc --build --watch running separately. Would be great to have project references resolve and build with ts-node alone.

[nickzelei]

FYI it looks like they've added API support for building project references.

https://github.com/microsoft/TypeScript/pull/31432

interface SolutionBuilder<T extends BuilderProgram> {
    build(project?: string, cancellationToken?: CancellationToken): ExitStatus;
    clean(project?: string): ExitStatus;
    buildReferences(project: string, cancellationToken?: CancellationToken): ExitStatus;
    cleanReferences(project?: string): ExitStatus;
    getNextInvalidatedProject(cancellationToken?: CancellationToken): InvalidatedProject<T> | undefined;
}

[krzkaczor]

FYI: the obvious workaround is to set TS_NODE_TRANSPILE_ONLY=true but then you won't get any type checks.

[dchambers]

@krzkaczor, I tried setting this on the command line (using --transpile-only and -T) and it makes no difference -- project references still aren't built.

[dchambers]

I ended up using concurrently to build the project references for all the libraries within our monorepo (all inside a lib directory) like so:

"scripts": {
  "start": "concurrently \"tsc -b -w ../lib\" \"nodemon -w ./src -w ../lib --ignore 'lib/*/src/*' -e js,jsx,gql,ts,tsx --exec ts-node src/server.ts\"",
}

Here, the lib directory contained a tsconfig.json with references to all the composite projects within the directory.

[calvinwyoung]

I'd also like to use ts-node with project references. Until this is supported, I'm using the following workaround:

tsc-watch -b --onSuccess 'node dist/index.js'

On my project, this picks up changes slightly faster than using concurrently with tsc and nodemon.

[Sytten]

Any progress on it?

[JasonKleban]

https://github.com/TypeStrong/ts-node/blob/f77e1b14a540ef50970adca16809547a1db3bce7/src/index.ts#L921

This comment is confusing me. Does it mean to say that there is a --build flag in ts-node? I can't find it in the code or the documentation. I assume it's just an artifact of a work in progress? (I'm trying to use naked pnpm rather than lerna, but I don't think that makes much difference.)

Here is a ~hidden pending change to the typescript compiler API wiki: https://github.com/microsoft/TypeScript-wiki/pull/225/files#diff-709351cd55688fbcb7ec0fc9973ee746R407

I am trying to figure out adding project references support to ts-node and then ts-node-dev, but I have low confidence that I can figure it out as a beginner contribution. If any of the maintainers have explored this before, can they please share their findings so far?

My basic guess is that we need a new CLI flag (--solution/--S) that basically replaces createIncrementalCompilerHost with createSolutionBuilderHost here:

https://github.com/TypeStrong/ts-node/blob/f77e1b14a540ef50970adca16809547a1db3bce7/src/index.ts#L787-L799

If it were that simple, it'd probably be done by now. I think that SolutionBuilderHost used to extend CompilerHost, but no longer does. CompilerHost rolls up to ModuleResolutionHost whereas SolutionBuilderHost rolls up to ProgramHost.

BTW, where is it that the compiled in-memory script is executed? I don't have any ideas of what to do after ts.createSolutionBuilder(host, Array.from(rootFileNames), {}).build(); Seems like I need to get a Program out of there somehow?

@sheetalkamat might know exactly what to do.

[cspotcode]

https://github.com/TypeStrong/ts-node/blob/f77e1b14a540ef50970adca16809547a1db3bce7/src/index.ts#L921

This comment is confusing me. Does it mean to say that there is a --build flag in ts-node? I can't find it in the code or the documentation. I assume it's just an artifact of a work in progress? (I'm trying to use naked pnpm rather than lerna, but I don't think that makes much difference.)

--build is referring to the incremental flag in tsconfig.json: if(options.emit && config.options.incremental) options.emit refers to ts-node's --emit flag/option, and config.options.incremental refers to TypeScript's incremental tsconfig option.

Here is a ~hidden pending change to the typescript compiler API wiki: https://github.com/microsoft/TypeScript-wiki/pull/225/files#diff-709351cd55688fbcb7ec0fc9973ee746R407

I am trying to figure out adding project references support to ts-node and then ts-node-dev, but I have low confidence that I can figure it out as a beginner contribution. If any of the maintainers have explored this before, can they please share their findings so far?

My basic guess is that we need a new CLI flag (--solution/--S) that basically replaces createIncrementalCompilerHost with createSolutionBuilderHost here:

I think we can get away with enabling this behavior by default, or at least enabling by default when --files is enabled. We might need to make --files default behavior in the future anyway.

BTW, where is it that the compiled in-memory script is executed? I don't have any ideas of what to do after ts.createSolutionBuilder(host, Array.from(rootFileNames), {}).build(); Seems like I need to get a Program out of there somehow?

Here is where we hook into node's module loading mechanism: https://github.com/TypeStrong/ts-node/blob/f77e1b14a540ef50970adca16809547a1db3bce7/src/index.ts#L1022-L1049

Node internally reads the file's contents from disk, then passes it to _compile. We wrap the _compile function to take the source text, compile it, and pass the emitted output to node's native _compile implementation, which handles module execution.

[JasonKleban]

Isn't typescript's --build CLI flag more related to composite: true and project references (v3.0) while incremental and tsBuildInfo (v3.4) can be used regardless of whether there's a typescript project-reference involved or not?

https://www.typescriptlang.org/docs/handbook/project-references.html#caveats-for-project-references says:

to preserve compatibility with existing build workflows, tsc will not automatically build dependencies unless invoked with the --build switch.

which I don't really understand - but why wouldn't ts-node need to respect this opt-in behavior?

Has this been reattempted since the API was made public to the extent that it is currently, and with the knowledge of the documentation in the wiki PR? I didn't see any branches here for it. Just wondering what trees the community can avoid barking up.

[cspotcode]

The following answers are based on memory. They might be wrong, they're not perfectly written, and they're long, but I tried to provide as much detail as possible.

Isn't typescript's --build CLI flag more related to composite: true and project references (v3.0) while incremental and tsBuildInfo (v3.4) can be used regardless of whether there's a typescript project-reference involved or not?

Correct, --build / composite imply and require incremental. If I had to guess, the comment you see in our source code about --build is conflating the two. The comment is potentially confusing; I wouldn't focus on it too much.

why wouldn't ts-node need to respect this opt-in behavior?

That's a good question. If the user runs a script in projectA which imports a file from projectB, what should we do? Should we follow project references and compile the file in projectB, using projectB's compiler options, essentially defaulting to --build mode? Should we eagerly type-check the entirety of projectB?

Also, ts-node has its own ignore option which determines which files we do/don't compile. Also, is having our own ignore option a good or bad idea?

To get more detailed, tsc and node load files in different ways, and we need to somehow bridge the gap.

TypeScript has the luxury of eagerly loading all projects and files, type-checking and emitting them all at once. It starts with a single tsconfig.json. It follows all project references, globs for all "files"/"includes", recursively parses all import statements, and adds those files, too. This process pulls more and more projects and files into the compilation, and all must be compiled and emitted. tsc's job is to eagerly pull all files into memory, parse and typecheck them all, and emit .js for all.

node, on the other hand, loads files on-demand when require()d or imported. When this happens, ts-node must ensure that tsc has already looked at the file. If it hasn't, ts-node forcibly adds it to the "files" array, triggering tsc to parse, type-check, and emit it. The language service is well-suited to this on-demand style of compilation, but I'm not sure on-demand fits well with project references.

I use the term tsc loosely to refer to the various TypeScript APIs we use.

Possible simplifications

This all needs research, but here are a few things that might simplify ts-node and help us support project references:

• make --files on by default and/or required for project references
• Disallow "forcibly adding" files as described above. If a file is not included in your "files" or "include" array, we throw a helpful error

About incremental and tsbuildinfo

I believe that implementing project references with the performance benefits of incremental will require writing all output into a separate, ts-node private cache. Related to #1161

ts-node uses potentially different compiler options than tsc, so our emitted output may look different. It can't be written to disk in the same place as tsc's output. For example, the user might specify TS_NODE_COMPILER_OPTIONS. Also, we override certain sourcemap options.

[cspotcode]

I realized I should clarify something:

That's a good question. If the user runs a script in projectA which imports a file from projectB, what should we do? Should we follow project references and compile the file in projectB, using projectB's compiler options, essentially defaulting to --build mode? Should we eagerly type-check the entirety of projectB?

What I mean by this is, if you import a file from projectB, then node is going to have a require() that needs to happen. What do we do when that file from projectB is require()d?

[cspotcode]

@JasonKleban I've been thinking about a good place for a beginner to start contributing to ts-node. I think #1161 is a great starting point. All contributions are appreciated, so of course you can focus your efforts anywhere you want. But if you're looking for something straightforward that will bring us closer to project references, #1161 is a great option.

[theomessin]

Bump

[vamcs]

I also have this problem and it would be really nice to avoid running the extra tsc --watch.

[theomessin]

This is a bit of a hack but I've written a small script ts-builder that can be used with Mocha to do tsc --build before loading the compiled JavaScript. It will emit the files as configured in tsconfig.json. This is what I'm now using with Mocha by doing:

mocha -r @theomessin/ts-builder/register **.test.ts

Should do until ts-node supports project references.

[wunderdaz]

This thread is really hard to find... Been on the hunt for hours on this issue.

It's a really necessary feature

[cspotcode]

Do you want to try your hand at writing a pull request for it? Check out #1514 and see if there's anything you want to help with.

Alternatively, if your employer is willing to pay for the feature to be prioritized, we might be able to work something out. We've not done that sort of thing before but we could give it a try.

[wunderdaz]

https://gitlab.com/darren-open-source/mo-ts-scaffold

If it helps at all, I setup a basic TS project with modular entrypoints... I've created a list of objectives (Which were based on many TS complaints across multiple forums/tickets) and have basically resolved most of them besides this issue.

Happy for anyone to check it out and try get things working. Hoping to create a good minimum viable TS (For node) project so developers have a great starting point.

I've found that, unfortunately, for any decent project there needs to be a build pipeline. I didn't go towards babel, as I prefer webpack to just put everything into one bundle file... This also solves a lot of potential deployment and production reference issues.

[zomars]

I've just stumbled upon this very same issue. I couldn't find any info on the best possible way to work with ts-node inside a monorepo. Tried using both "require": ["tsconfig-paths/register"] and "experimentalResolverFeatures": true with no luck. :/

[zachkirsch]

+1 I would also love a way of running ts-node on a cli that lives in a monorepo (and imports from other monorepo packages using tsconfig references)

[softmarshmallow]

Essential for developing monorepo cli apps. +1. I have no solid solutions for this problem yet, but I'll share my progress here with my production project - https://github.com/gridaco/cli

[zomars]

This is what we're doing at @calcom

https://github.com/calcom/cal.com/blob/d1d467d28d03811b50ec2942c8eeef82e2e425a5/packages/tsconfig/base.json#L20-L28

[nicoabie]

https://github.com/calcom/cal.com/blob/d1d467d28d03811b50ec2942c8eeef82e2e425a5/packages/tsconfig/base.json#L20-L28

thanks @zomars !!!

I was having issues with a custom typing and the files: true flag solved the issue.

This is my tsconfig.json of the test folder that references the source and has to reference a custom typing the source uses:

{
  "extends": "../tsconfig.json",
  "compilerOptions": {
    "composite": true
  },
  "include": ["./", "../types/index.d.ts"],
  "references": [{ "path": "../src" }],
  "ts-node": {
    "files": true,
  }
}