esbuild
esbuild copied to clipboard
evanw
How to fix "Dynamic require of "os" is not supported"
How can we use dependencies that still use require while shipping esm? Is that possible?
Minimal reproduction: https://github.com/enricoschaaf/esbuild-issue Just run node index.mjs. This was generated from running build which executes esbuild to bundle this file.
Problem: The problem is that node internal modules can of course not bet bundled so I expected the require syntax would get written to import syntax if the target is esm but they are still require.
Error message: Error: Dynamic require of "os" is not supported at file:///home/enrico/work/esbuild-issue/index.mjs:1:450 at file:///home/enrico/work/esbuild-issue/index.mjs:1:906 at file:///home/enrico/work/esbuild-issue/index.mjs:1:530 at file:///home/enrico/work/esbuild-issue/index.mjs:1:950 at ModuleJob.run (node:internal/modules/esm/module_job:195:25) at async Promise.all (index 0) at async ESMLoader.import (node:internal/modules/esm/loader:337:24) at async loadESM (node:internal/process/esm_loader:88:5) at async handleMainPromise (node:internal/modules/run_main:65:12)
As far as I know, esbuild cannot convert external "static" commonjs require statements to static esm imports - not even for the side effect free node platform built in modules. This would be a useful feature for esbuild to have, as it's a fairly common use case for NodeJS applications and libraries. One could write an esbuild plugin for that, but a comprehensive solution would require reparsing each JS source file and the build would be considerably slower.
Ok, I thought about writing one but I had the same reservations you had. But indeed that would be a super nice feature for node apps.
Although this functionality would be better in esbuild itself, here's a super hacky script to transform node built-in __require()s in esbuild esm output to import statements. It only works with non-minified esbuild output.
$ cat importify-esbuild-output.js
var fs = require("fs");
var arg = process.argv[2];
var data = !arg || arg == "-" ? fs.readFileSync(0, "utf-8") : fs.readFileSync(arg, "utf-8");;
var rx = /\b__require\("(_http_agent|_http_client|_http_common|_http_incoming|_http_outgoing|_http_server|_stream_duplex|_stream_passthrough|_stream_readable|_stream_transform|_stream_wrap|_stream_writable|_tls_common|_tls_wrap|assert|async_hooks|buffer|child_process|cluster|console|constants|crypto|dgram|diagnostics_channel|dns|domain|events|fs|http|http2|https|inspector|module|net|os|path|perf_hooks|process|punycode|querystring|readline|repl|stream|string_decoder|sys|timers|tls|trace_events|tty|url|util|v8|vm|wasi|worker_threads|zlib)"\)/gm;
var modules = new Map;
var out = data.replace(rx, function(req, mod) {
var id = "__import_" + mod.toUpperCase();
modules.set(mod, id);
return id;
});
modules.forEach(function(val, key) {
console.log("import %s from %s;", val, JSON.stringify(key));
});
console.log("\n%s", out);
Example input:
$ cat 0.js
console.log(require("path").extname("foo/bar.hello"));
var os = require("os");
console.log(require("path").extname("ok.world"));
Expected output:
$ cat 0.js | node
.hello
.world
Unmodified esbuild esm bundle output:
$ cat 0.js | esbuild --bundle --platform=node --format=esm
var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function(x) {
if (typeof require !== "undefined")
return require.apply(this, arguments);
throw new Error('Dynamic require of "' + x + '" is not supported');
});
// <stdin>
console.log(__require("path").extname("foo/bar.hello"));
var os = __require("os");
console.log(__require("path").extname("ok.world"));
...piped through importify-esbuild-output.js:
$ cat 0.js | esbuild --bundle --platform=node --format=esm | node importify-esbuild-output.js
import __import_PATH from "path";
import __import_OS from "os";
var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function(x) {
if (typeof require !== "undefined")
return require.apply(this, arguments);
throw new Error('Dynamic require of "' + x + '" is not supported');
});
// <stdin>
console.log(__import_PATH.extname("foo/bar.hello"));
var os = __import_OS;
console.log(__import_PATH.extname("ok.world"));
... and then piped through esbuild again to perform some DCE:
$ cat 0.js | esbuild --bundle --platform=node --format=esm | node importify-esbuild-output.js | esbuild --bundle --platform=node --format=esm
// <stdin>
import __import_PATH from "path";
console.log(__import_PATH.extname("foo/bar.hello"));
console.log(__import_PATH.extname("ok.world"));
to produce:
$ cat 0.js | esbuild --bundle --platform=node --format=esm | node importify-esbuild-output.js | esbuild --bundle --platform=node --format=esm | node --input-type=module
.hello
.world
In addition to the workaround above:
esbuild actually will print a warning when converting require to esm format (see it live) like:
▲ [WARNING] Converting "require" to "esm" is currently not supported
<stdin>:1:0:
1 │ require('a')
╵ ~~~~~~~
Maybe we can use that info to write a plugin: https://gist.github.com/hyrious/7120a56c593937457c0811443563e017
One note: the modification actually turns a commonjs module into mixed module, however it seems esbuild can handle this case correctly:
import xxx from 'xxx'
module.exports = xxx
I'm getting this with require("buffer/") and from what I can gather, libs use the trailing slash to bypass core module and lookup in node_modules folder. I believe this is because they provide the polyfill for the browser as part of the package.
For now, I've added @esbuild-plugins/node-globals-polyfill and @esbuild-plugins/node-modules-polyfill, and then patched the packages to use require("buffer") so that they reference the esbuild polyfill instead but wondering if esbuild should be able to handle trailing slashes here?
This is all a bit over my head tbh, so apologies in advance if I'm not being clear 😅
require("buffer/") is a different issue and appears to be working as intended - see https://github.com/evanw/esbuild/commit/b2d7329774fd9c42cb5922a9b5825498b26a28f3.
@kzc in my case, it seems the trailing slash is being used to lookup a buffer polyfill for the browser so i am getting this error client-side when --platform=browser.
@jjenzz If you're seeing a bug you should probably open a separate issue for that with a reproducible test case, as it doesn't appear to be related to this issue. This issue is concerned with --format=esm --platform=node bundles producing require() instead of import for NodeJS built-in modules. Your issue appears to be related to a NodeJS module polyfill not being inlined into your bundle for --format=esm --platform=browser.
$ rm -rf node_modules/buffer
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm | node --input-type=module
✘ [ERROR] Could not resolve "buffer/"
<stdin>:1:20:
1 │ console.log(require("buffer/"));
╵ ~~~~~~~~~
You can mark the path "buffer/" as external to exclude it from the bundle, which will remove this
error. You can also surround this "require" call with a try/catch block to handle this failure at
run-time instead of bundle-time.
1 error
$ mkdir -p node_modules/buffer
$ echo "module.exports = {abc: 42};" > node_modules/buffer/index.js
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm | node --input-type=module
{ abc: 42 }
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm
var __getOwnPropNames = Object.getOwnPropertyNames;
var __commonJS = (cb, mod) => function __require() {
return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
};
// node_modules/buffer/index.js
var require_buffer = __commonJS({
"node_modules/buffer/index.js"(exports, module) {
module.exports = { abc: 42 };
}
});
// <stdin>
console.log(require_buffer());
Notice that require("buffer/") was inlined as expected once node_modules/buffer/ was in place. node --input-type=module would have errored out had it encountered a require().
The trailing backslash did not appear to matter for the browser platform if node_modules/buffer existed - identical esbuild output was produced:
$ cat node_modules/buffer/index.js
module.exports = {abc: 42};
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm --platform=browser | shasum
a4d2d5027a1ab792dcf6bfb6f0ddc63e273a2b11 -
$ echo 'console.log(require("buffer"));' | esbuild --bundle --format=esm --platform=browser | shasum
a4d2d5027a1ab792dcf6bfb6f0ddc63e273a2b11 -
$ echo 'console.log(require("buffer"));' | esbuild --bundle --format=esm --platform=browser | node --input-type=module
{ abc: 42 }
$ echo 'console.log(require("buffer/"));' | esbuild --bundle --format=esm --platform=browser | node --input-type=module
{ abc: 42 }
@kzc thanks for all of that. I'll see if I can create a minimal repro of the issue I am experiencing and create a separate issue if so.
Related: https://stackoverflow.com/questions/68423950/when-using-esbuild-with-external-react-i-get-dynamic-require-of-react-is-not-s?rq=1 It has no solution either
To share some details about impact, it looks like this is:
- blocking Netlify's migration to ESM: https://answers.netlify.com/t/netlify-serverless-typescript-functions-fail-with-dynamic-requires/52379/13
- blocking Shopify from dropping CJS support: https://github.com/Shopify/hydrogen/issues/1155
FYI, I created a PR that addresses this issue for Node built-ins specifically: https://github.com/evanw/esbuild/pull/2067
@eduardoboucas @jjenzz @kzc @enricoschaaf hey do you have any idea why this same Dynamic require of {module} is not supported error is coming up even when I've specified --format=iife --platform=browser in my esbuild command? My webpage uses a file that has the umd snippet at the beginning, and trying to minify it with esbuild causes this error to come up. I can see that esbuild is inserting a check for require at the start of the file, but I'm not sure why or how to prevent that. My full command is
esbuild test/**/*.js --bundle --minify --tree-shaking=false --format=iife --platform=browser --external:@most --outdir=./build
and the full error is Dynamic require of "@most/prelude" is not supported
Any ideas?
This is breaking on AWS lambda with crypto module
{
"errorType": "Error",
"errorMessage": "Dynamic require of \"crypto\" is not supported",
"trace": [
"Error: Dynamic require of \"crypto\" is not supported",
" at file:///var/task/dbStreamHandler.js:29:9",
" at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/rng.js (file:///var/task/dbStreamHandler.js:10354:42)",
" at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
" at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/v1.js (file:///var/task/dbStreamHandler.js:10439:39)",
" at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
" at ../node_modules/.pnpm/[email protected]/node_modules/uuid/dist/index.js (file:///var/task/dbStreamHandler.js:10822:37)",
" at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
" at ../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js (file:///var/task/dbStreamHandler.js:10930:18)",
" at __require2 (file:///var/task/dbStreamHandler.js:44:50)",
" at ../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-retry/dist-cjs/AdaptiveRetryStrategy.js (file:///var/task/dbStreamHandler.js:11023:35)"
]
}
any workarounds?
@ShivamJoker we have a dependency that references os but doesn't get used in our code. We worked around it with this in our package.json:
{
"browser": {
"os": "os-browserify",
},
}
@ShivamJoker Have you found a solution for this? Experiencing the same issue with AWS lambda.
For AWS Lambda with serverless-esbuild plugin this config work for me
esbuild:
minify: true
platform: node
target: esnext
sourcemap: true
sourcesContent: false
format: esm
outExtension:
.js: .mjs
banner:
js: import { createRequire } from 'module';const require = createRequire(import.meta.url);
`
@fospitia can you please share more details about how you are using this config with esbuild?
@ShivamJoker the cause of the bug is that function 'require' not exists in an ES module of nodejs. The banner option add the 'require' function at the beginning of generated file.
If you use the serverless framework with the serverles-esbuild plugin use below config.
esbuild:
minify: true
platform: node
target: esnext
sourcemap: true
sourcesContent: false
format: esm
outExtension:
.js: .mjs
banner:
js: import { createRequire } from 'module';const require = createRequire(import.meta.url);
With esbuid API you canuse the BuildOptions below
const buildOptions = {
bundle: true,
minify: true,
format: "esm",
target: "esnext",
platform: "node",
banner: {
js: 'import { createRequire } from 'module';const require = createRequire(import.meta.url);'
},
outExtension: {
.js: '.mjs'
}
};
esbuild(buildOptions);
This configs work with AWS Lambda with node16 runtime.
Thanks @fospitia I got it to work. Here is my full script file which I am importing in CDK
const { build } = require("esbuild");
const { sync } = require("fast-glob");
const { join } = require("path");
const paths = sync("**/*.ts", {
cwd: join(process.cwd(), "./lambda"),
absolute: true,
});
// console.log(paths);
console.log("Bundling lambdas to ESM modules");
build({
absWorkingDir: process.cwd(),
bundle: true,
logLevel: "info",
entryPoints: paths,
outdir: join(process.cwd(), "./lambda/dist"),
minify: true,
format: "esm",
target: "esnext",
platform: "node",
banner: {
js: "import { createRequire } from 'module';const require = createRequire(import.meta.url);",
},
outExtension: {
".js": ".mjs",
},
})
.catch(() => process.exit(1));
@ShivamJoker the cause of the bug is that function 'require' not exists in an ES module of nodejs. The banner option add the 'require' function at the beginning of generated file.
If you use the serverless framework with the serverles-esbuild plugin use below config.
esbuild: minify: true platform: node target: esnext sourcemap: true sourcesContent: false format: esm outExtension: .js: .mjs banner: js: import { createRequire } from 'module';const require = createRequire(import.meta.url);With esbuid API you canuse the BuildOptions below
const buildOptions = { bundle: true, minify: true, format: "esm", target: "esnext", platform: "node", banner: { js: 'import { createRequire } from 'module';const require = createRequire(import.meta.url);' }, outExtension: { .js: '.mjs' } }; esbuild(buildOptions);This configs work with AWS Lambda with node16 runtime.
thanks man, saved us !!!
it worked without addtional import
import { build } from "esbuild";
/** @type {import('esbuild').BuildOptions} */
const options = {
minify: true,
bundle: true,
target: "node16",
banner: {
js: `
import { createRequire } from 'module';
import path from 'path';
import { fileURLToPath } from 'url';
const require = createRequire(import.meta.url);
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
`
},
platform: "node",
format: "esm",
};
const funcDir = ["HttpTrigger1"]
for (const dir of funcDir) {
/** @type {import('esbuild').BuildOptions} */
const opts = {
...options,
entryPoints: [`./${dir}/index.ts`],
outfile: `./dist/${dir}/index.mjs`,
}
build(opts).catch((err) => {
process.stderr.write(err.stderr);
process.exit(1);
});
}
I think esbuild could automatically add --banner:js="import {createRequire} from 'module';const require=createRequire(import.meta.url);" when --platform is node. It's needed for pretty much all dependencies that require node core modules.
I don't know where else to put this same but I'm getting this error with react while attempting to build a component library. I'm using tsup and this is my config file
import { defineConfig } from "tsup";
export default defineConfig((options) => {
return {
entry: ["src/index.ts"],
format: ["cjs", "esm"],
dts: true,
external: ["react"],
minify: !options.watch
};
});
Removing the external entry fixes the problem but that's not an option.
Adding a banner as someone suggested above did not fix the issue and instead it brought out another error saying
✘ [ERROR] Expected "}" but found "file"
tsup.config.ts:12:80:
12 │ ...obal.require = createRequire("file:///home/myusername/apath...
│ ~~~~
╵ }
I can't find anything else about these errors so I have no idea how to fix them...
I have a local script file that I want to import but I am getting the same error. I am not sure why it must be so difficult to import some external script using the import keyword in the browser. The functionality is already built into the browser.
For ESM, node16, this one should address the problem well, even when combining builds with the "same" banner and including support for handling subsequent import/requires in 3rd party libs of user code and such cases...:
import { BuildOptions } from "esbuild";
/** quasi-random identifier generator, good for less than 10k values in series */
export const getRandomBase32Identifier = (length: number = 9) =>
`i${Math.random().toString(36).substring(2, length)}`
export const getEsmCjsCompatOptions = (): BuildOptions => {
const dirnameAlias = getRandomBase32Identifier();
const pathToFileURLAlias = getRandomBase32Identifier();
const createRequireAlias = getRandomBase32Identifier();
const esmCjsCompatScript = `
import { pathToFileURL as ${pathToFileURLAlias} } from 'url';
import { createRequire as ${createRequireAlias} } from 'module';
import ${dirnameAlias} from 'es-dirname';
import.meta.url = ${pathToFileURLAlias}(${dirnameAlias}());
const require = ${createRequireAlias}(import.meta.url);
`
return {
target: 'node16',
platform: "node",
format: "esm",
banner: {
js: esmCjsCompatScript
}
}
}
This implementation makes sure that when userland code imports dirname, etc. top-level, it wouldn't break with "duplicate identifier" error, as all identifiers are randomly generated, which is an issue with the import {createRequire} from 'module';const require=createRequire(import.meta.url); approach. Someone imports "createRequire" in userland code and it breaks. Also: depending on the runtime, import.meta.url won't be defined. That's why the 3rd party module es-dirname is used, which uses a hack to retrieve the dirname by parsing an artificially thrown Node.js stacktrace... (yeah yeah.. I know... what the heck...)
Apply it to your esbuild bundle config as you like:
import { build } from "esbuild"
build({
...getEsmCjsCompatOptions(),
...your other options here...
})
This should not be closed. I had to hack it like this:
const sharedConfig =
{
entryPoints: ['src/main/cli.ts'],
bundle: true,
//minify: true,
sourcemap: 'linked',
target: ['esnext'],
platform: 'node',
external: Object.keys(dependencies),
}
console.log('compile start')
// build the code - esm style
console.log(buildSync({
...sharedConfig,
format: 'esm',
outfile:'dist/main.esm.js',
banner: {
js:`import { createRequire } from 'module';const require = createRequire(import.meta.url);`
}
}))
I couldn't believe it, but it actually solved the problem.
There should be a parameter that says:
makeItWorkInNode:true
that does this without me having to blindly hack like this. Aside from this, absolutely love the tool!
I think a good "fix" for this issue would be to show the offending dependency. Adding that dependency to external is the next step to solve the problem for the user.
Folks using AWS JS SDK V2 ("aws-sdk") running into this might consider switching to the V3 JS SDK (e.g. "@aws-sdk/client-dynamodb") as a solution
(I realize there is work in that, just wanted to mention it as an option)
