vitejs
Cannot bundle .node files
Describe the bug
I am trying to create a server side bundle that includes packages containing .node files. In this specific instance, the problematic package is nodejs-polars.
in/nodejs-polars.linux-arm-gnueabihf.node does not exist
✓ 13 modules transformed.
✓ built in 1.39s
[commonjs] Unexpected character '' (Note that you need plugins to import files that are not JavaScript)
file: /home/projects/vitejs-vite-xp1ggj/node_modules/nodejs-polars/bin/series/list.js:1:0
1: ELF>�rd@X�@8
^
2: @@@0�bd�bd...
3:
I have tried various plugins to resolve this but none of them seem to work.
My config is as follows:
import native from 'vite-plugin-native';
export default function () {
return {
ssr: {
noExternal: ['nodejs-polars'],
},
optimizeDeps: {
include: ['linked-dep', 'node_modules'],
exclude: ['nodejs-polars'],
},
esbuild: {
platform: 'node',
target: 'node16',
},
build: {
lib: {
entry: './main.js',
name: 'demo',
},
ssr: true,
target: 'node16',
},
plugins: [native({ target: 'esm' })],
};
}
Reproduction
https://stackblitz.com/edit/vitejs-vite-xp1ggj?file=vite.config.ts,package.json&terminal=dev
Steps to reproduce
Run vite build --config vite.config.ts
System Info
System:
OS: Linux 5.4 Ubuntu 20.04.6 LTS (Focal Fossa)
CPU: (8) x64 AMD FX(tm)-8350 Eight-Core Processor
Memory: 9.66 GB / 31.31 GB
Container: Yes
Shell: 5.8 - /usr/bin/zsh
Binaries:
Node: 18.14.2 - ~/.nvm/versions/node/v18.14.2/bin/node
Yarn: 1.22.19 - ~/.nvm/versions/node/v18.14.2/bin/yarn
npm: 8.9.0 - ~/.nvm/versions/node/v18.14.2/bin/npm
Browsers:
Brave Browser: 115.1.56.14
Chrome: 116.0.5845.96
### Used Package Manager
npm
### Logs
_No response_
### Validations
- [X] Follow our [Code of Conduct](https://github.com/vitejs/vite/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitejs/vite/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitejs.dev/guide).
- [X] Check that there isn't [already an issue](https://github.com/vitejs/vite/issues) that reports the same bug to avoid creating a duplicate.
- [X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to [vuejs/core](https://github.com/vuejs/core) instead.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitejs/vite/discussions) or join our [Discord Chat Server](https://chat.vitejs.dev/).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.
![bolt-new-by-stackblitz[bot] avatar](https://avatars.githubusercontent.com/in/182710?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue is preventing me from using many modules. No workaround or package has worked.
Well, that's one hell of a hack. But I'll have to study and use it I guess. Thank you so much. Resolving .node files should be a feature in Vite.
The same problem occurs with fsevent.
I have confirmed that it works by adding the following settings to astro.config.mjs.
vite: {
optimizeDeps: {
exclude: ["fsevents"],
},
},
https://stackoverflow.com/questions/75640753/vite-esbuild-error-no-loader-is-configured-for-node-files-node-modules-fs
Error log
13:51:15 [vite] Forced re-optimization of dependencies
astro v4.4.9 ready in 341 ms
┃ Local http://localhost:3001/
┃ Network use --host to expose
13:51:15 watching for file changes...
✘ [ERROR] No loader is configured for ".node" files: node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.node
node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.js:13:23:
13 │ const Native = require("./fsevents.node");
╵ ~~~~~~~~~~~~~~~~~
13:51:25 [200] / 40ms
13:51:25 [ERROR] [UnhandledRejection] Astro detected an unhandled rejection. Here's the stack trace:
Error: Build failed with 1 error:
node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.js:13:23: ERROR: No loader is configured for ".node" files: node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.node
at failureErrorWithLog ({...}/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1651:15)
at {...}//node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1059:25
at {...}//node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1527:9
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Hint:
Make sure your promises all have an `await` or a `.catch()` handler.
Error reference:
https://docs.astro.build/en/reference/errors/unhandled-rejection/
Stack trace:
at {...}//node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.js: ERROR: No loader is configured for ".node" files: node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.node:13:23
[...] See full stack trace in the browser, or rerun with --verbose.
I managed to bundle nodejs-polars using this config but it's incredibly hacky.
do you have any idea how to handle this problem of canvas library, i am using nuxt3 and vite but can't handle it, thank you
vite.optimizeDeps.exclude solution doesn't work with pnpm workspaces. Example https://github.com/stereobooster/braindb
pnpm build
...
demo-astro-cli:build: 20:25:22 [vite] Re-optimizing dependencies because vite config has changed
demo-astro-cli:build: 20:25:23 [build] output: "static"
demo-astro-cli:build: 20:25:23 [build] directory: /mddb/packages/demo-astro-cli/dist/
demo-astro-cli:build: 20:25:23 [build] Collecting build info...
demo-astro-cli:build: 20:25:23 [build] ✓ Completed in 1.09s.
demo-astro-cli:build: 20:25:23 [build] Building static entrypoints...
demo-astro-cli:build: 20:25:25 [ERROR] [vite] x Build failed in 1.85s
demo-astro-cli:build: [commonjs--resolver] ../../node_modules/.pnpm/[email protected]/node_modules/fsevents/fsevents.node (1:0): Unexpected character '�' (Note that you need plugins to import files that are not JavaScript)
demo-astro-cli:build: file: /mddb/node_modules/.pnpm/[email protected]/node_modules/chokidar/index.js:1:0
demo-astro-cli:build: 1: ����@<�
��*...
demo-astro-cli:build: ^
4*mo-astro-cli:build: 2:
h���/System/Library/Frameworks/CoreFoundation.framework/Versions...
demo-astro-cli:build: 3: ���H��1�H���L���L���H�����L�E�L��H��1�����uH�E�H�[A^A_]��}f.�UH��AWAVSPI��H��I��H�H...
demo-astro-cli:build: Stack trace:
demo-astro-cli:build: at getRollupError (file:///mddb/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:379:41)
demo-astro-cli:build: at convertNode (file:///mddb/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:12914:10)
demo-astro-cli:build: at Module.setSource (file:///mddb/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:14073:24)
demo-astro-cli:build: Caused by:
demo-astro-cli:build: Unexpected character '�'
demo-astro-cli:build: at getRollupError (file:///mddb/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/parseAst.js:379:41)
demo-astro-cli:build: at convertNode (file:///mddb/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:12914:10)
demo-astro-cli:build: at Module.setSource (file:///mddb/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:14073:24)
demo-astro-cli:build: ELIFECYCLE Command failed with exit code 1.
demo-astro-cli:build: ERROR: command finished with error: command (/mddb/packages/demo-astro-cli) pnpm run build exited (1)
demo-astro-cli#build: command (/mddb/packages/demo-astro-cli) pnpm run build exited (1)
Are there any plans on this one? Even though @goastler shared a nice blog, it doesn't solve anything for the general public.
The handling of .node files (and other binary-like files) which occur in nodejs packages needs to be addressed by either vite or a plugin. If vite will not support this (as it could be considered out of scope) then it will have to be a plugin. Our business interests have moved away from this at the moment, so making a plugin would be very low priority for @forgetso and myself at Prosopo.
Further, there's more complications with files other than .node files (iirc .glyph) which serve similar purpose, so we'd need to do a bunch of research into how to handle those files correctly.
tldr; probably better for someone who knows more about these files to implement a plugin / build support in vite
For those searching for a solution, I (chatgpt to be honest) came up with a quick plugin:
function nativeFilesPlugin(): PluginOption {
const files = new Map<string, { readonly fileName: string; readonly fileContent: Buffer }>();
return {
name: 'node-binaries-plugin',
async load(id) {
if (!id.endsWith('.node')) {
return null;
}
const fileContent = await fs.readFile(id);
const hash = createHash('sha256').update(fileContent).digest('hex').slice(0, 8);
const fileName = `${path.basename(id, '.node')}.${hash}.node`;
files.set(id, { fileName, fileContent });
return `export default require('./${fileName}');`;
},
generateBundle(_, bundle) {
for (const [id, { fileName, fileContent }] of files.entries()) {
this.emitFile({ type: 'asset', fileName, source: fileContent });
delete bundle[id];
}
},
};
}
I'm building into cjs, so I also have these settings:
build: {
rollupOptions: {
output: {
format: 'cjs',
},
},
commonjsOptions: {
requireReturnsDefault: 'auto',
}
}
It looks like an even simpler version also works:
{
name: 'require-node-binaries-plugin',
transform: (code, id) => {
if (id.endsWith('.node')) {
return code.replace(/export default "(.+)"/, 'export default require(".$1")');
}
return;
},
}
paired with build.ssrEmitAssets: true and assetsInclude: ['**/*.node']
Small update here.
Native assets didn't work in my project because I'm using uWebSockets.js, which is an ESM-wrapper around CJS-module that does dynamic requires. No matter how hard I tried, I wasn't able to make them work with assets. Dynamic requires are typically handled by @rollup/plugin-commonjs which is a whole different story, as it does some code transformations that are hard to address, if vite treats .node binaries as assets.
So I had to revert to the original naive plugin, and, unfortunately, add a special hard-coded case for uWebSockets.js because of dynamic requires.
function nativeFilesPlugin(): PluginOption {
interface NativeFile {
readonly fileName: string;
readonly fileContent: Buffer;
}
const nativeFiles = new Map<string, NativeFile>();
const uws = require.resolve('uWebSockets.js');
const readNativeFile = async (filePath: string): Promise<NativeFile> => {
const fileContent = await readFile(filePath);
const hash = createHash('sha256').update(fileContent).digest('hex').slice(0, 8);
const fileName = `${path.basename(filePath, '.node')}.${hash}.node`;
return { fileName, fileContent };
};
return {
name: 'native-files-plugin',
async load(id) {
if (id === uws) {
// Special handling for the ESM-wrapper around CJS-module with dynamic requires (uWebSockets.js).
const nativeFile = await readNativeFile(
// Yes, build the file name at build time, not runtime 🤷♂️(we can't do dynamic `import {} from ''`)
path.resolve(path.dirname(uws), './uws_' + process.platform + '_' + process.arch + '_' + process.versions.modules + '.node'),
);
nativeFiles.set(id, nativeFile);
return `export default require('./${nativeFile.fileName}')`;
} else if (id.endsWith('.node')) {
const nativeFile = await readNativeFile(id);
nativeFiles.set(id, nativeFile);
return `export default require('./${nativeFile.fileName}');`;
}
return;
},
generateBundle(_, bundle) {
for (const [id, { fileName, fileContent }] of Array.from(nativeFiles.entries())) {
this.emitFile({ type: 'asset', fileName, source: fileContent });
delete bundle[id];
}
},
};
}
I believe the issue should be handled on rollup's side, not on vite's, as I believe it's purely rollup's job to bundle everything into a module graph that vite would understand. There's also an on-going work on Rolldown (a Rust replacement for rollup), so maybe it should even go there, or maybe it's already supported there, I can't test it right now.
In my case, everything was solved after deleting package-lock.json and node_modules and installing everything again.
I have many native deps and its works fine, but sqlite3 / better-sqlite3 have some different code to load .node files.
In my case, fully solved by "rollup-plugin-natives": "^0.7.8"
I am trying to import a binary .node file on my project, but these solutions failed, any suggestions?
After searching for a while, I wrote https://github.com/biw/vite-plugin-native-modules
