Meta: Native support for ES Modules

[SimenB]

EDIT: quick guide for getting started: https://jestjs.io/docs/ecmascript-modules

ESM support will be unflagged in a future release of Node 12 (maybe not before April https://github.com/nodejs/node/pull/29866#issuecomment-574055057) and it is already unflagged in Node 13.2, so I think it's time to evaluate how we can add native support in Jest. I'll try to list which features Jest currently provides that are impacted by ESM support, and how we can solve/investigate them.

There is issue #4842, but I think that's more of a discussion issue, while this issue will be geared towards actually implementing support and more suitable to track for those who just want to get the current implementation status. Any comments added to this issue not related to how we can implement support for the below enumerated features will be marked as spam - please direct any workarounds/discussions to separate issues. Also feel free to tell us if anything related to ESM features is missing from the list!

Please note that Jest will use the vm API (https://nodejs.org/api/vm.html) and as of writing (node ~v13.6~ v16.10) the ESM parts of this API is still flagged (--experimental-vm-modules). So saying ESM is unflagged is a bit of a misnomer at the moment. But I think we should start experimenting and potentially provide feedback to the Modules WG.

EDIT: Tracking issue for stabilization in Node: https://github.com/nodejs/node/issues/37648

Lastly, I'm writing this issue mostly for people who will implement support, so it'll be somewhat low-level and specific to how Jest works. For people who just want to know whether support has landed or not, I recommend using GH's wonderful "custom notification" and only subscribe to notifications on closing/reopening.

---

• [x] Running the module in the correct context

We achieve sandboxes by running a script within a given vm.Context (either provided by JSDOM or node core APIs). We need to do the same for ESM, but we'll need access to the context during construction of the module, not just when executing the module. I've opened up #9428 which adds the necessary APIs to JestEnvironment.

• [x] Globals

expect, test, beforeEach etc will still be added as globals, nothing should change here. jasmine global will also still be here.

• [x] jest "global" property

This is not really a global - it's injected into the module scope. Since the module scope is gone in ESM, we need to move it somewhere. Adding it to import.meta seems natural - there's an option called initializeImportMeta which we can use.

EDIT: Solution here is to fetch it via import {jest} from '@jest/globals'. We might still add it via import.meta in the future, but this should be enough for now.

• [ ] jest.(do|un)mock

Since ESM has different "stages" when evaluating a module, jest.mock will not work for static imports. It can work for dynamic imports though, so I think we just have to be clear in the docs about what it supports and what it doesn't.

jest.mock calls are hoisted, but that doesn't help in ESM. We might consider transforming import 'thing' to import('thing') which should allow hoisting to work, but then it's async. Using top-level await is probably a necessity for such an approach. I also think it's invasive enough to warrant a separate option. Something to discuss - we don't need to support everything jest.mock can for for an initial release.

PR: #10976

• [ ] jest.requireActual

Not sure if how it should behave in ESM. Should we provide a jest.importActual and let requireActual evaluate in CJS always?

• [x] import.meta

Node has url as its only property (for now, at least). We need to make sure it's populated in Jest as well. We provide identifier instead of filename when constructing the module so I don't think it'll happen automatically, but url is essentially filename passed though pathToFileURL.

There's also an open PR for import.meta.resolve: https://github.com/nodejs/node/pull/31032

• [x] import thing from 'thing'

This should actually be fairly straightforward, we just need to implement a linker where we can also transform the source before returning it, meaning we don't need the loader API (which doesn't exist yet). This allows us to return mocks as well (albeit they'll have to come from a __mocks__ directory).

• [x] import('thing')

Essentially the same as above, but passed as importModuleDynamically when constructing the module. Will also support jest.mock, jest.resetModules etc more cleanly, so likely to be used quite a bit.

This can also be done for vm.Script via the same option.

• [ ] Handling errors during evaluation

Right now it's a runtime error (e.g. module not found), but that's not necessarily true with ESM. Does it matter for us? We should verify errors still look nice.

• [x] module.createRequire

We need to deal with this for people wanting to use CJS from ESM. I've opened up #9426 to track this separately as implementing it is not really related to ESM support.

EDIT: Implemented in #9469

• [ ] module.syncBuiltinESMExports

https://nodejs.org/api/modules.html#modules_module_syncbuiltinesmexports. Do we care about it, or is just making it a no-op enough? Not sure what the use case in Jest would be. Messing with the builtins is already breaking the sandbox and I don't think this should matter.

EDIT: #9469 made this into a no-op. I think that's fine?

• [x] Detect if a file is supposed to be ESM or CJS mode

Inspecting type field in a module's package.json seems reasonable: https://nodejs.org/api/esm.html#esm_enabling. Should we also have our own config flag? Also needs to respect file endings.

https://github.com/nodejs/node/issues/49446

• [x] moduleNameMapper

Not sure if this impacts anything. I think not since we'll be linking the modules together ourselves. Needs investigation, though.

EDIT: This is all resolution logic, which we control. So no changes here.

• [x] jest.config.mjs

Through #9291 we support jest.config.cjs - do we need to do anything special for .mjs? Probably use import('path/to/configFile.mjs') which means it'll have to be async. Is this an issue? Might be worth making config resolution async in Jest 25 so it's not a blocker for incremental support of ESM in Jest 25.

EDIT: #9431

• [x] Package Exports

Node supports package exports, which sorta maps to Jest's moduleNameMapper, but also provides encapsulation features. Hopefully resolve will implement this, but if they do not we'll need to do something. Might be enough to use the pathFilter option? Unsure.

EDIT: #9771

• [ ] JSON/WASM module

https://nodejs.org/api/esm.html#esm_experimental_json_modules. Do we need to care? Probably, especially for json. It's trivial for us to support import thing from './package.json' since we control the linking phase, but we probably shouldn't do it by default as it'll differ from default node. Should we force people to define a transform for it?

WASM: #13505

• [x] Code coverage

Does it matter? I don't think it's affected as we can still transform the source with babel (maybe it'll be confused by import statements, probably not) and V8 coverage definitely shouldn't care. We should verify though.

• [x] Async code resolution

This is absolutely no blocker as sync resolution will work just fine. But we can use async resolution now, which is great. I wonder if we should look into just using the resolve module off of npm again, as it already supports async. See #9505.

• [x] Async code transformation

Similar to above, not blocking, but would be nice to support it. Might make @jest/transformer more usable in other environments as well. See #9504.

EDIT: #9889 & #11191

• [ ] Bad performance when accessing globals

Due to #5163 we have the extraGlobals option as a workaround - that workaround is no longer viable in ESM. I've opened up and issue with node here: https://github.com/nodejs/node/issues/31658

• [x] Import assertions

https://nodejs.org/api/esm.html#import-assertions

[SimenB]

I've landed very basic support with #9772. I've only tested the simplest cases, and there are many known limitations (most notably no jest object support and broken semantics when mixing CJS and ESM), but at least it's something. It'll go out in the next release of Jest (hopefully soon, only blocked by #9806)

[SimenB]

25.4.0 has been released with the first pieces of support. In addition to #9772 mentioned above, I've also included #9842. In theory mixing CJS and ESM should work correctly now (🤞).

The one main missing feature is supporting the jest object. I haven't decided if we should stick it to import.meta or require people to import it through import {jest} from '@jest/globals'. Feedback appreciated!

I haven't written docs for this yet, but to activate it you need to do 3 things

1. make sure you don't run transform away import statements (set transform: {} in config or otherwise ensure babel doesn't transform the file to CJS, such as avoiding the modules option to preset-env)
2. Run node@^12.16.0 || >=13.2.0 with --experimental-vm-modules flag
3. Run your test with jest-environment-node or jest-environment-jsdom-sixteen

Please try it out and provide feedback! If reporting bugs, it'd be wonderful if you can also include how running the same code (minus any test specific code) runs in Node. I've read https://nodejs.org/api/esm.html a lot over the last few weeks, but I've probably missed something.

[just-boris]

The one main missing feature is supporting the jest object. I haven't decided if we should stick it to import.meta or require people to import it through import {jest} from '@jest/globals'.

For the typescript use-case it is better to have an explicit import.

[SimenB]

Yup, I've added (and the temporarily reverted) a @jest/globals package that supports this, so that will be available regardless. I'm wondering if it makes sense to also expose it on import.meta. Currently leaning towards not doing so, mainly since it's easier to add than remove later (and I'm personally no fan of globals)

[IlCallo]

+1 for the explicit import, it's a bit more verbose but simpler to understand

[zandaqo]

I'm getting this in Node 13.2 & Jest 25.4: ES Modules are only supported if your test environment has the getVmContext function What am I missing?

[SimenB]

@zandaqo Oh sorry, forgot that point. Added above, but it's

Run your tests with jest-environment-node or jest-environment-jsdom-sixteen

[zandaqo]

ReferenceError: jest is not defined I'm guessing this is due to the missing @jest/globals

[SimenB]

Yes, as mentioned this will only work if you don't use the jest object. Mocks are also probably broken, haven't tested them 😃

[s-ilya]

I've compiled a very basic project from what I see in e2e tests directory (e2e/native-esm/__tests__/native-esm.test.js) and in this issue. And unfortunately I still can't make it work 🙃Can anybody check it by any chance?

https://drive.google.com/file/d/1vyDZjsVKOTu6j55QA11GjO9E7kM5WX8_/view?usp=sharing

• [x] jest version 25.4.0
• [x] node version v13.12.0
• [x] package.json contains recommended jest options and no babel transformations seem to be in place

Running sample script (just importing double function and printing double(2)):

npm run main

> [email protected] main /Users/ilya/Projects/jest-esm
> node src/main.js

(node:16961) ExperimentalWarning: The ESM module loader is experimental.
4

Running jest with just one test for double function:

npm run test

> [email protected] test /Users/ilya/Projects/jest-esm
> jest

 FAIL  __tests__/native-esm.test.js
  ● Test suite failed to run

    Jest encountered an unexpected token

    This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.

    By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".

    Here's what you can do:
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/en/configuration.html

    Details:

    /Users/ilya/Projects/jest-esm/__tests__/native-esm.test.js:8
    import {double} from '../src/index.js';
    ^^^^^^

    SyntaxError: Cannot use import statement outside a module

      at Runtime._execModule (node_modules/jest-runtime/build/index.js:1074:58)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        0.542s
Ran all test suites.

[SimenB]

You need to run node with --experimental-vm-modules and either name you file .mjs or "type": "module" in package.json.

EDIT: You probably have the latter seeing as it works outside of Jest for you, just missing the experimental flag to Node

[s-ilya]

@SimenB oh wow there are --experimental-vm-modules AND --experimental-modules. I was confused by the fact that --experimental-modules is not required starting from some node 13 version. Thanks!

TLDR: node --experimental-vm-modules node_modules/jest/bin/jest.js works for me

[SimenB]

Yeah, from the OP

Please note that Jest will use the vm API (https://nodejs.org/api/vm.html) and as of writing (node v13.6) the ESM parts of this API is still flagged (--experimental-vm-modules). So saying ESM is unflagged is a bit of a misnomer at the moment.

Awesome that it works for you, though!

(I'll mark these comments as resolved)

[aldeed]

@SimenB Thanks for this! Two issues I'm seeing so far.

Issue 1

• ESM unit test file imports default from ESM file (fn being tested)
• ESM file being tested imports default from a package, which exports CJS only
• Results in error: ReferenceError: module is not defined in the CJS package file

So I'm thinking this may not be correctly implemented?

An import statement can reference an ES module or a CommonJS module

This works fine when running the app. Named exports from CJS can only be imported by using createRequire, but default exports can just be imported.

Issue 2

When I'm not hitting the above error, I'm hitting this one:

TypeError: _vm(...).SyntheticModule is not a constructor

      at Runtime._importCoreModule (node_modules/jest-runtime/build/index.js:1198:12)

Project details

• Node 12.14.1
• Jest and babel-jest 25.4.0
• Latest Babel with targets: { node: 12 } and 2 plugins: babel-plugin-transform-import-meta and rewire-exports. (Tried removing import-meta plugin but got an error saying to add it back.)
• testEnvironment: "node" is pretty much the only config
• node --experimental-vm-modules node_modules/jest/bin/jest.js

If a reproduction repo would be helpful, let me know.

[SimenB]

Thanks @aldeed!

I'll look into issue 1, that indeed looks like a bug. EDIT: Should be fixed via #9850

Issue 2 needs node 12.16.0: https://nodejs.org/docs/latest-v12.x/api/vm.html#vm_class_vm_syntheticmodule

I'll change the check in Jest (right now it checks for vm.SourceTextModule which is available in more versions, but we need SyntheticModule as well).

[aldeed]

Confirmed that running with 12.16.0 fixes my issue 2. Will retest issue 1 after that fix is released. Otherwise waiting on the jest object for further testing, and I agree that it should need to be imported.

[beejunk]

Awesome work, @SimenB! I'm trying this out on a small project but ran into issues with dynamic imports. This is the error I'm seeing:

Module status must not be unlinked or linkingError [ERR_VM_MODULE_STATUS]: Module status must not be unlinked or linking

If I remove the dynamic imports then the test will run (but fail for other reasons, of course). The same test is currently working with Mocha (which shipped ESM support fairly recently).

If it helps, the dynamic imports in question can be seen here: https://github.com/beejunk/firn.js/blob/switch-to-jest/lib/renderPage.js#L43-L55

The test file is here: https://github.com/beejunk/firn.js/blob/switch-to-jest/test/build.test.js. The working Mocha version can be seen on the master branch.

Let me know if there's any other info that my be useful.

[SimenB]

Thanks @beejunk! I've been wondering if there could be race conditions between imports that import the same module before it's fully linked. It seems like that's what your hitting here. I'll fix this today, thanks for the report!

EDIT: Fixed in #9858. Copied the fix over to your repo:

[dandv]

Has anyone managed to get this to work with TypeScript? node --experimental-vm-modules node_modules/jest/bin/jest.js returns the same SyntaxError: Cannot use import statement outside a module, even though my package.json does have "type": "module".

babel.config.cjs

module.exports = {
  presets: [
    '@babel/preset-typescript',
  ],
};

jest.config.cjs

module.exports = {
  testEnvironment: 'jest-environment-node',
  transform: {},
};

[SimenB]

@dandv You're essentially hitting this case: https://github.com/facebook/jest/pull/9772/files#r407255029

Could you open up a separate issue? Will need to figure out what to do with non-js extensions

[dandv]

@SimenB: #9860. Thanks for taking a look.

[SimenB]

@aldeed @beejunk 25.5.0 has been released with fixes for your issues. Keep the bug reports coming 😀

[SimenB]

Oh, additionally it includes support for import { jest } from '@jest/globals' for those of you waiting for that 👍

[beejunk]

Thanks for quick work on all this, @SimenB! I think I've run into another issue.

I've been experimenting with using query params in an import path as a way of busting the module cache on a development server. The idea is to have a file watcher that detects changes in a component and then updates the import path with an arbitrary query param so that the new code is immediately pulled in on the next page load without having to restart the server. This works when running the dev server. However, Jest is throwing the following error:

Cannot find module '/path/to/project/components/BasePage.js?cache=0' from 'renderPage.js'

Here's an example of where the query params are being used: https://github.com/beejunk/firn.js/blob/switch-to-jest/lib/renderPage.js#L55-L56

If I remove the query params then the tests will pass, although not consistently. If I run a single suite (for example, npm test -- test/build.test.js) the tests pass, but if I run all the tests at once they will fail most of the time with ambiguous errors. I'm still digging into the issue with the inconsistent tests, but I thought I'd report on the query param problem first.

[SimenB]

Thanks for the report @beejunk. #6282 is supposed to deal with this, but that also wants to pass the query to transformers and stuff, that we don't need here. So it might make sense to just support the query internally in the runtime for now, and let #6282 just deal with passing that query on.

I've added a bit of code to make the module cache vary by query: https://github.com/facebook/jest/blob/d425a49bd575e7167bc786f3c4f2833589091fa1/packages/jest-runtime/src/index.ts#L330-L334

But no code ever calls it with a query. I think we should be able to just do resolvePath.split('?') and all should work.

Regarding inconsistent errors, I'll take a look if that repo reproduces it. I haven't tested the ESM code with tests in parallel, only a single test. I'm unsure why it would impact things, but who knows 😀

[SimenB]

@beejunk query thing fixed in 25.5.1. I haven't had time to investigate the other issue yet

[nahueld]

I have a problem that I think may be related to this however didn't get fix in 25.X.

I'll try to summarise the escenario below:

• You have a test with a setup script
• That setup script will require a file dynamically, as in: { default: generator } = require(path.resolve(f))
• Everything inside f is not transpiled causing the "unexpected identifier import error".

This also happens if I try with import()

[SimenB]

Since you mention transpilation; if you have a setup that transpiles import to require this issue is not the correct place - this issue is about native support.

---

That said - you cannot require ESM, and I haven't been able to add support for import() from CJS yet due to Node having no API for it. Support for that has landed on Node master, but not released yet: https://github.com/nodejs/node/pull/32985. As can be seen in the linked release PRs, it'll come in 13.14.0 and 14.1.0. At that point I'll implement support for it 👍

You should be able to use a .mjs setup file though.

[aldeed]

@SimenB This is great, it seems to be working in several test files now! It's a bit of a process to resolve the differences between Babel and Node imports in each file, add jest import, etc., so I may come across more issues as I do that across hundreds of test files.

A few things that are more questions:

• what you said in your previous comment about support for import() from cjs, will that also allow for the Jest config file to be named jest.config.js? It currently only works for me when named jest.config.cjs and using module.exports = (error is TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified)
• The name "@jest/globals" fails the eslint rule node/no-extraneous-import due to not being a package listed in package-lock. Is there some reason for this naming convention? Is it possible to use jest without the @? It's easy to ignore the rule but I'm just wondering.
• Related question, why is jest global different from magic fns like test,describe, before, etc.? Should all of those be imported now, too?
• When this is finalized will it be possible for Jest to set --experimental-vm-modules flag? It kind of makes this seem nonstandard if it doesn't just work with the jest command. If you're not able to directly pass flags, possibly you can set/amend the NODE_OPTIONS env variable?
• I now seem to get a slightly different error than before when trying this on older Node, but neither error was clear that minimum Node version was the problem. Is it easy to add a check for minimum version with a more helpful error message?

[SimenB]

will that also allow for the Jest config file to be named jest.config.js?

It can be named .js if its closest package.json has type: 'module'. Otherwise to use ESM in the config file, it needs to be named .mjs. See https://nodejs.org/api/esm.html#esm_enabling. Note that the config file runs outside of Jest, so there we don't have much control. At the moment, we don't support async config, but awaiting an exported promise would be trivial, PR welcome 🙂 #8357 has stalled.

I'm very surprised you get A dynamic import callback was not specified though, we don't load the config file in a vm... Could you open up a separate issue?

The name "@jest/globals" fails the eslint rule node/no-extraneous-import due to not being a package listed in package-lock. Is there some reason for this naming convention? Is it possible to use jest without the @? It's easy to ignore the rule but I'm just wondering.

You should add a devDependency on @jest/globals. The package itself is solely type definitions. It's a separate package from jest so that type definitions work correctly, and so we can throw an error if it's loaded outside of the Jest runtime.

I don't currently have any plans to change that, but we could possibly deprecate that package and export the types from jest. That's a major breaking change though, so let's see later 🙂

Related question, why is jest global different from magic fns like test,describe, before, etc.? Should all of those be imported now, too?

jest is like require or module object from CJS in that it's unique per file, it's not actually a global (i.e. globalThis.jest === undefined). That allows jest.mock('../../file.js') etc to work correctly relative to the file it was injected in. You can choose to also import the actual globals, but they are still available as globalThis.expect etc.

When this is finalized will it be possible for Jest to set --experimental-vm-modules flag?

I think we'll wait for node to unflag them rather than silently set them - they're flagged for a reason, the API might change at some point. We could add an option that set NODE_OPTIONS I guess, not sure if that changes the current runtime? Regardless, no current plans for it.

I now seem to get a slightly different error than before when trying this on older Node, but neither error was clear that minimum Node version was the problem. Is it easy to add a check for minimum version with a more helpful error message?

Yeah, I'll be adding a better error message along with some documentation once the implementation stabilizes a bit. Seeing as the implementation is guarded by an experimental flag, I don't think anyone will stumble over it.