ZJONSSON
Build fails without @aws-sdk/client-s3 package in package.json
Hey!
When I install the unzipper npm package and don't install separately @aws-sdk/client-s3 I get the build exception like below.
Please move @aws-sdk/client-s3 to required dependencies for automatic installation, or fix it usage.
Thanks in advance
ERROR in ./node_modules/unzipper/lib/Open/index.js 97:52-81
Module not found: Error: Can't resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
Parsed request is a module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./lib/Open)
resolve as module
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open\node_modules doesn't exist or is not a directory
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\node_modules
single file module
using description file: E:\Work\finsemble-mono\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\node_modules doesn't exist or is not a directory
E:\node_modules doesn't exist or is not a directory
@ ./node_modules/unzipper/unzip.js 5:0-36
@ ./src/main/ExternalApplicationManager.ts 4:0-32 70:27-43
@ ./src/main/Main.ts 4:0-74 120:43-69
@ ./src/app.ts 12:0-35 100:17-21
ERROR in E:\Work\finsemble-mono\packages\electron-adapter\src\main\ExternalApplicationManager.ts
33:20-40
[tsl] ERROR in E:\Work\finsemble-mono\packages\electron-adapter\src\main\ExternalApplicationManager.ts(33,21)
TS2307: Cannot find module '@aws-sdk/client-s3' or its corresponding type declarations.
ts-loader-default_04bb2ec108ef140d
webpack 5.91.0 compiled with 2 errors and 2 warnings in 18167 ms
To keep node-unzipper super small, a decision was made to not include optional third party sdks as a part of the library itself. unzipper has plenty of users that do not require the s3 features and it would be very inefficient to force them to do so (example: https://www.npmjs.com/package/exceljs)
If you want to use the aws_s3 you will have to ensure that the sdk is installed, i.e. npm install unzipper @aws-sdk/client-s3, perhaps the right thing to do is clarify this in the README
@MaksimDedov can you confirm the build error only occurs because you are actually use the aws_v3 in the build process?
@ZJONSSON In my code I use unzipper in the next way and only once:
const unzipperStream = unzipper.Extract({ path: downloadAsset.assetFolder });
fileStream.pipe(unzipperStream);
I don't call any other functions. The downloadAsset.assetFolder is the path to local folder on machine, fileStream is Node.js transform stream. As I understood the Extract uses aws_v3 somewhere under the hood in ./node_modules/unzipper/lib/Open/index.js 97:52-81
The build error occurs if I don't add the aws_v3 in my package.json near unzipper
Thank that is very helpful. This seems to be a typescript/ts-loader compilation error not a runtime error. Seems like the appropriate fix here is a simple @ts-ignore.
unfortunately I see the same issue for [email protected]:
ERROR in ./node_modules/unzipper/lib/Open/index.js 98:52-81
Module not found: Error: Can't resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
Parsed request is a module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./lib/Open)
resolve as module
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open\node_modules doesn't exist or is not a directory
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\node_modules
single file module
using description file: E:\Work\finsemble-mono\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\node_modules doesn't exist or is not a directory
E:\node_modules doesn't exist or is not a directory
@ ./node_modules/unzipper/unzip.js 5:0-36
@ ./src/main/ExternalApplicationManager.ts 4:0-32 70:27-43
@ ./src/main/Main.ts 4:0-74 120:43-69
@ ./src/exports.ts 31:24-46
webpack 5.91.0 compiled with 1 error and 2 warnings in 19696 ms
Same error on my side ^^"
"unzipper": "^0.12.3"
Stack trace from Next:
Module not found: Can't resolve '@aws-sdk/client-s3'
https://nextjs.org/docs/messages/module-not-found
Import trace for requested module:
./node_modules/unzipper/unzip.js
./src/server/actions/files.ts
./src/server/data/mods.ts
./node_modules/next/dist/build/webpack/loaders/next-flight-action-entry-loader.js?actions=%5B%5B%22E%3A%5C%5CGitHub%5C%5Cfaithful%5C%5Cwebsite%5C%5Csrc%5C%5Cserver%5C%5Cdata%5C%5Cmods.ts%22%2C%5B%22getModsWithVersions%22%2C%22deleteMod%22%2C%22updateModPicture%22%2C%22voidMods%22%2C%22getMods%22%2C%22getModsFromIds%22%2C%22getModWithModVersions%22%2C%22updateMod%22%2C%22createMod%22%2C%22modHasUnknownVersion%22%5D%5D%2C%5B%22E%3A%5C%5CGitHub%5C%5Cfaithful%5C%5Cwebsite%5C%5Csrc%5C%5Cserver%5C%5Cdata%5C%5Cmods-version.ts%22%2C%5B%22getModVersionProgression%22%2C%22createModVersion%22%2C%22getSupportedMinecraftVersions%22%2C%22removeModpackFromModVersion%22%2C%22getModsVersionsFromResources%22%2C%22updateModVersion%22%2C%22deleteModVersion%22%2C%22getModVersionsWithModpacks%22%2C%22getNumberOfTextureFromModVersion%22%2C%22getModVersions%22%2C%22addModVersionsFromJAR%22%5D%5D%5D&__client_imported__=true!
GET / 500 in 1049ms
GET /_next/static/webpack/a1548000c16fc3ef.webpack.hot-update.json 500 in 1028ms
⚠ Fast Refresh had to perform a full reload due to a runtime error.
And the only thing I'm doing with unzipper is the following:
const bytes = await jar.arrayBuffer(); // where jar is a File
const buffer = Buffer.from(bytes);
const archive = await unzipper.Open.buffer(buffer);
Similar issue when using "unzipper": "^0.12.3" and NextJS
> next build
▲ Next.js 14.2.5
- Environments: .env
Creating an optimized production build ...
Failed to compile.
./node_modules/unzipper/lib/Open/index.js
Module not found: Can't resolve '@aws-sdk/client-s3'
https://nextjs.org/docs/messages/module-not-found
Temporary workaround: Add client-s3 to devDependencies
npm i -D @aws-sdk/client-s3@latest
I'm getting this error in my vue app
My Code This is the only instance where I'm using the library, and it builds fine without it.
const directory = await Unzipper.Open.buffer(newFile)
Terminal Error
13:29:14 [vite] hmr update /src/pages/media/add/media.vue, /src/components/media/AddFiles.vue (x2)
✘ [ERROR] Could not resolve "@aws-sdk/client-s3"
node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/Open/index.js:98:60:
98 │ const { GetObjectCommand, HeadObjectCommand } = require('@aws-sdk/client-s3');
╵ ~~~~~~~~~~~~~~~~~~~~
You can mark the path "@aws-sdk/client-s3" as external to exclude it from the bundle, which will
remove this error and leave the unresolved path in the bundle. You can also surround this
"require" call with a try/catch block to handle this failure at run-time instead of bundle-time.
13:29:17 [vite] error while updating dependencies:
Error: Build failed with 1 error:
node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/Open/index.js:98:60: ERROR: Could not resolve "@aws-sdk/client-s3"
at failureErrorWithLog (C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\[email protected]\node_modules\esbuild\lib\main.js:1651:15)
at C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\[email protected]\node_modules\esbuild\lib\main.js:1059:25
at C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\[email protected]\node_modules\esbuild\lib\main.js:1527:9
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
UPDATE 1 I tried adding in the error handling as described by the message in my terminal and it got me past the error. Unfortunately I'm getting some other errors now:
TypeError: util.inherits is not a function
at node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/PullStream.js (PullStream.js:18:6)
at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
at node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/parse.js (parse.js:4:20)
at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
at node_modules/.pnpm/[email protected]/node_modules/unzipper/unzip.js (unzip.js:2:17)
at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
at unzip.js:5:16
Is Unzipper not designed to work within a non-server environment or something? If it's any help, I'm using Node v20.14.0 and I'm compiling to ESNext
UPDATE 2
util is a node specific module that is not available in the browser. A step towards solving this is to polyfill those modules. As my project uses vite, this is best achieved with a package like https://github.com/davidmyersdev/vite-plugin-node-polyfills#vite-plugin-node-polyfills. That said, this still didn't quite work for me, so I'm going to try a browser first package like jszip.
This package does look really nice though!
@ZJONSSON
it is a good idea to keep the package small. but since you are using require then it failed to bundle or build in the absence of the optional packages
i recommend to use something like "optional-require" so it wont fail and you can check the dependency you need before using it
This is a bundling issue. Bundler can't tell at build time whether the dependency will be used or not, so tries to bundle it but it's not installed.
For those using Nextjs (or a webpack based framework), something like the following will work as an alternative to installing the dependency:
const nextConfig: NextConfig = {
webpack: (config) => {
config.resolve.alias = {
...config.resolve.alias,
"@aws-sdk/client-s3": false,
};
return config;
}
Edit: added return config to example.
@ZJONSSON how about having different entry points for this? Something like:
import { Open } from `unzipper/s3`
I think you can do this within the one package by splitting out the code and modifying your package.json:
"name": "unzipper",
"exports": {
".": "./unzip.js",
"./s3": "./lib/Open/s3.js",
}
}
Alternatively listing @aws-sdk/client-s3 as a peerDependency might generate more helpful logs.
PS: great package, thanks for shipping! ❤️
I am also facing the problem Error: Cannot find module '@aws-sdk/client-s3' with vite and version 0.12.3
"@aws-sdk/client-s3" is imported by "@aws-sdk/client-s3?commonjs-external", but could not be resolved – treating it as an external dependency.
⚠️ rollback to version 0.11.6 fixed the problem
@rnbrady Unfortunately that doesn't seem to work. Our next.config.ts includes this:
webpack: (config) => {
config.resolve.alias = {
...config.resolve.alias,
'@aws-sdk/client-s3': false,
'@aws-sdk/s3-presigned-post': false,
}
},
But results in these warnings still:
⚠ ./node_modules/.pnpm/@[email protected][email protected]/node_modules/@aws-sdk/s3-presigned-post/dist-es
Package @aws-sdk/client-s3 can't be external
The request @aws-sdk/client-s3 matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @aws-sdk/client-s3 from the project directory.
⚠ ./packages/file/src
Package @aws-sdk/client-s3 can't be external
The request @aws-sdk/client-s3 matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @aws-sdk/client-s3 from the project directory.
⚠ ./packages/file/src
Package @aws-sdk/s3-presigned-post can't be external
The request @aws-sdk/s3-presigned-post matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @aws-sdk/s3-presigned-post from the project directory.
@luskin it's a function so you need to return the modified config object:
const nextConfig: NextConfig = {
webpack: (config) => {
config.resolve.alias = {
...config.resolve.alias,
"@aws-sdk/client-s3": false,
};
return config;
}
Apologies for the typo in original suggestion, which I've now corrected.
The following would also work and is the safer way to write it:
const nextConfig: NextConfig = {
webpack: (config) => ({
...config,
resolve: {
...config.resolve,
alias: {
...config.resolve.alias,
"@aws-sdk/client-s3": false,
},
},
}),
};
I tested both of these with Next.js 15.1.4 and [email protected] in next dev (without turbo) and next build.
To keep node-unzipper super small, a decision was made to not include optional third party sdks as a part of the library itself.
This is not a correct way of shipping the package. As it can be seen from the comments above, the issue affects many people in very basic scenarios that don't involve any "advanced" AWS features.
A correct way would be splitting out the package into two: the "minimal" one called unzipper and an addtional one called something like unzipper/with-aws-stuff. The unzipper/with-aws-stuff package would import the "minimal" unzipper and then somehow "hook" into it via some kind of a plugin API or just wrap it with additional pre- or post- processing code, etc.
P.S. We encountered this error when using esbuild bundler.
One workaround is to install "@aws-sdk/client-s3" package manually, which would still lead to increased bundle size. If the code is bundled for server-side-use only, that is will not be used in a web browser, then the increased bundle size would not be an issue.
Another workaround could be to "alias" "@aws-sdk/client-s3" package in a "bundler" configuration file with a path to a *.js file containing just "export default null". But that kind of a workaround would also result in errors when using other packages that import anything from "@aws-sdk/client-s3" package, so it's not really a workaround but more of a ticking bomb.
I ended up with the following ignore rule inside my webpack config:
plugins: [
new webpack.IgnorePlugin({ resourceRegExp: /^@aws-sdk\/client-s3$/ }),
],
