Node 20 auto-loader not able to transpile TS to JS when using ts-node/esm

[TomasHubelbauer]

Prerequisites

• [X] I have written a descriptive issue title
• [X] I have searched existing issues to ensure the bug has not already been reported

Fastify version

4.10.0

Plugin version

No response

Node.js version

20.1.0

Operating system

macOS

Operating system version (i.e. 20.04, 11.3, 10)

13.3.1

Description

I think this is a Node issue, but only @fastify/autoloader is affected by it, so I am opening an issue with you good folks as well in case this is really a Fastify issue instead or if it is a Node issue but you are able to find a workaround!

  fastify.register(fastifyAutoload, {
    dir: join(__dirname, 'routes'),
    forceESM: true,
  });

This piece of code works in Node 19 when using node --loader=ts-node/esm (the route files are in TypeScript) but not on Node 20.

I have come up with a repro here: https://github.com/TomasHubelbauer/node-esm-loader-repro

Steps to Reproduce

Follow my repro repo: https://github.com/TomasHubelbauer/node-esm-loader-repro

Steps copied here for posterity.

Node 20:

1. nvm install 20 to install Node 20
2. node --version to ensure Node version (I get 20.1.0)
3. npm install to install dependencies
4. npm run test to run the health.test.ts script

Notice the test fails and Fastify's autoload is seemingly not aware of the --loader option and attempts to load routes/health.ts without TypeScript to JavaScript conversion via ts-node/esm.

npm run test

> [email protected] test
> node --loader=ts-node/esm --experimental-specifier-resolution=node --test health.test.ts

ℹ (node:95105) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
ℹ (Use `node --trace-warnings ...` to show where the warning was created)
✖ should be alive (32.750836ms)
  Error: "@fastify/autoload cannot import plugin at '/routes/health.ts'. To fix this error compile TypeScript to JavaScript or use 'ts-node' to run your app."
      at findPlugins (/node_modules/@fastify/autoload/index.js:224:15)
      at async autoload (/node-esm-loader-repro/node_modules/@fastify/autoload/index.js:35:22)

ℹ tests 1
ℹ suites 0
ℹ pass 0
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 2208.335483

✖ failing tests:

✖ should be alive (32.750836ms)
  Error: "@fastify/autoload cannot import plugin at 'routes/health.ts'. To fix this error compile TypeScript to JavaScript or use 'ts-node' to run your app."
      at findPlugins (/node_modules/@fastify/autoload/index.js:224:15)
      at async autoload (/node_modules/@fastify/autoload/index.js:35:22)

Node 19:

1. nvm install 19 to install Node 20
2. node --version to ensure Node version (I get 19.9.0)
3. npm install to install dependencies
4. npm run test to run the health.test.ts script

Notice the test passes and Fastify's autoload is inherit the --loader option and uses the ts-node/esm loader successfully to auto-load routes/health.ts.

npm run test

> [email protected] test
> node --loader=ts-node/esm --experimental-specifier-resolution=node --test health.test.ts

ℹ (node:95453) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
ℹ (Use `node --trace-warnings ...` to show where the warning was created)
✔ should be alive (472.711395ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 2679.742161

Expected Behavior

Works the same way in both 19 and 20.

[mcollina]

Moved to the autoload module.

[climba03003]

Works the same way in both 19 and 20.

I would just hold this issue and only fix it when --loader becomes stable. As far as I know, there are numbers of issue about the experimental loader. The error is mostly cause by https://github.com/nodejs/node/pull/44710

[mcollina]

The APIs for loaders changed in v20 and it broke ts-node, mostly due to https://github.com/nodejs/node/issues/47880.

Once the dust settle and there is an API for us to use, we'll need to make adaptation here.

[Joabesv]

Hi guys! I recently got bit by this, do we have any updates?

[mcollina]

Loaders are mostly stable now, so A PR would be highly welcomed, otherwise this will take a bit.

[Joabesv]

not saying that I can do it, but where in the code should I take a look to know about how autoload deal with the loader flag?

[Uzlopak]

I debugged it and it seems, that (Symbol.for('ts-node.register.instance') in process) is returning false. You can test it yourself:

clone the repo, open in vscode, install deps, set break point in node_modules/@fastifa/autoload/index.js in line 8.

run npm t in javascript debug terminal . In node 18 the symbol is set, in node 20 not.

[Uzlopak]

Possibly related to https://github.com/TypeStrong/ts-node/issues/1997

[Joabesv]

thank you for the very informative tips on how to test it @Uzlopak. was testing with node 20.6.0 and the problem happens with tsx too! I guess is happening with all ts-loaders

[Uzlopak]

But you could do it "manually" by setting e.g. the TS_NODE_DEV to a truthy value.

 "scripts": {
    "test": "TS_NODE_DEV=1 node --loader=ts-node/esm --experimental-specifier-resolution=node --test health.test.ts"
  },

Maybe we should add a environment variable, like FASTIFY_AUTOLOAD_TS to generally override it

[thaivlinh]

@Uzlopak 's tip work. In my case I'm using tsx.

"dev": "TS_NODE_DEV=1 node --no-warnings=ExperimentalWarning --watch --loader=tsx src/server.ts",

[Joabesv]

Can we label this issue as has workaround so others can see it has a solution?

[Uzlopak]

I created a PR #327

It is basically the same as TS_NODE_DEV but with the doubt, that it should not result in some side effects or whatever, because we use our own ENV variable.

[Joabesv]

since the pr is now merged, I believe this one can be closed? @Uzlopak @mcollina

[Uzlopak]

the original issue is still unresolved

[rhettjay]

I believe this is also preventing my tests from running when using autoload in my tests. https://github.com/jay-bulk/autoload-min Neither ts-node or tsx work (even with the workaround hard-coded). @Uzlopak any thoughts as to why the workaround doesn't work in the instance of node:test?

[climba03003]

@jay-bulk I would say your issue is not related here. Please use a proper tsconfig.json if you want to use ESM and your example is broken.

From my testing, either --loader ts-node/esm or --import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("ts-node/esm", pathToFileURL("./"));' works.

[rhettjay]

The min repo may be broken, but my code (upon which the min is based) is using a proper tsconfig and is experiencing the same issue. I'll have to try your longer form import. That aside, neither ts-node/esm nor tsx-esm worked in conjunction with the FASTIFY_AUTOLOAD_TYPESCRIPT env var set to true. Do you know of a way for me to test autoload's assumption that tsx is in my preloaded modules?

[rhettjay]

~/sandbox/autoload-min[main] npm run test

> [email protected] test
> FASITY_AUTOLOAD_TYPESCRIPT=1 node --import=ts-node/esm --test index.ts

TypeError: Unknown file extension ".ts" for /Users/rbulkley/sandbox/autoload-min/index.ts
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:160:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:203:36)
    at defaultLoad (node:internal/modules/esm/load:143:22)
    at async ModuleLoader.load (node:internal/modules/esm/loader:409:7)
    at async ModuleLoader.moduleProvider (node:internal/modules/esm/loader:291:45)
    at async link (node:internal/modules/esm/module_job:76:21) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

I've added a tsconfig to the min repo, which still produces the above result. Interestingly, with the tsconfig included and running tsx (without esm enforcement) I get the expected result. With esm enforced in tsx I get other errors. However, ts-node/esm is as shown above.

A few subsequent inspect tests make it appear as if the _preload_modules are not being set. I don't know enough about that env variable to know where I could be going wrong.

[rhettjay]

Testing the autoload forceEsm tests with node:test likewise fails. It feels like a node issue, but I'm out of my depth. I'll just import the routes manually in my tests.

[climba03003]

There is nothing called --import ts-node/esm only --loader ts-node/esm. You should read how to use TSNode for --require, --loader or --import.

diff --git a/diff b/diff
new file mode 100644
index 0000000..e69de29
diff --git a/package.json b/package.json
index 1c3c795..db6f838 100644
--- a/package.json
+++ b/package.json
@@ -5,7 +5,7 @@
   "main": "index.js",
   "type": "module",
   "scripts": {
-    "test": "node --import=ts-node --test index.ts"
+    "test": "node --loader=ts-node/esm --test index.ts"
   },
   "keywords": [],
   "author": "",
diff --git a/load/index.ts b/route/load/index.ts
similarity index 100%
rename from load/index.ts
rename to route/load/index.ts
diff --git a/tsconfig.json b/tsconfig.json
new file mode 100644
index 0000000..cb7f695
--- /dev/null
+++ b/tsconfig.json
@@ -0,0 +1,6 @@
+{
+  "compilerOptions": {
+    "module": "NodeNext",
+    "moduleResolution": "NodeNext"
+  }
+}

[climba03003]

I am going to lock the issue for collaborator only. Most of the solution is posted above, if anything is not working. It is due the configuration problem on your environment.

For example,

1. Messed up with ESM and tsconfig.json
2. Messed up with the usage of ts-node, etc.

For CJS user, You should run the command with --loader ts-node/esm and FASTIFY_AUTOLOAD_TYPESCRIPT=1

FASTIFY_AUTOLOAD_TYPESCRIPT=1 node --loader ts-node/esm <entrypoint>

For ESM user, You should run the command with --loader ts-node/esm, FASTIFY_AUTOLOAD_TYPESCRIPT=1 and VITEST=true

FASTIFY_AUTOLOAD_TYPESCRIPT=1 VITEST=true node --loader ts-node/esm <entrypoint>

[Uzlopak]

I got the request to review this lock and to potentially moderate by @rhettjay

After talking with @climba03003 I agree that the reported issue by @rhettjay is a misconfiguration in tsconfig, which results in a transpiled code by tsc so that nodejs cant load the code properly.

We can not fix this as it is a pure configuration problem.also i assume we can not detect such misconfigurations at runtime.

I checked and we are lacking this information in our README. So, @rhettjay , if you want you could add a new section in our Readme, which describes the necessary configuration for fastify autoload to work with typescript.

Regarding the lock of the issue in this case needs the consent of @climba03003 to lift it. Also the question is, if this is still an issue. Maybe documenting the limitation resolves this issue. Then we could close it for good.

IMHO if you take care of the documentation, we could then reevaluate this issue. If there is still an issue we have to decide what we do with it. If we document properly, what @climba03003 wrote, than the issue which you reported will become out of scope. So maybe then @climba03003 will agree on an unlock.

If sombody then reports similar problems we can redirect that person to the corresponding part of the documentation. If the issue gets heated again, we can lock it again.

Summary: The lock is staying. You can provide a PR documenting limitations of this plugin in an typescript environment. After this we can reevaluate this issue.

Personal Note: I dont think that you, @rhettjay , had a bad intent. But insisting on your report did not land well and we have not much resources to discuss things for days and weeks. Also every off topic answer potentially spams other github users subscribed to this issue. If you provide a PR with the mentioned changes you can show that you evaluated the answer of @climba03003 and probably everything will resolve happily.

Unfortunately i cant give you a better answer but I think this is answering your request to everybodies satisfaction.

Looking forward for your upcoming participation in OSS. :).

Best Regards Aras

[climba03003]

Please check the latest version 5.8.1 to see if it helps. It will currently detect the --loader ts-node/esm automatically and using import. So, it no longer requires FASTIFY_AUTOLOAD_TYPESCRIPT, VITEST or forceESM.

I will close the issue, but open public comment for feedback.