Bundling with Webpack

[Bessonov]

Possible install-time or require-time problem

• [x] I have read and understood all of the documentation relating to installation.
• [x] I have searched for known bugs relating to this problem in my choice of package manager.

You must confirm both of these before continuing.

Are you using the latest version of sharp?

I first tried with the latest version (0.34.1), but saw here some current issues, so now I'm trying 0.33.5 instead.

Are you using a supported runtime?

• [x] I am using Node.js with a version that satisfies ^18.17.0 || ^20.3.0 || >=21.0.0: v22.14.0

Are you using a supported package manager and installing optional dependencies?

• [x] I am using pnpm >= 7.1.0 with --no-optional=false: 10.6.5

What is the complete error message, including the full stack trace?

What is the complete output of running npm install --verbose --foreground-scripts sharp in an empty directory?

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

First of all, thank you for the amazing library!

I’m packaging my app using Webpack and generate a single server.mjs file along with the necessary *.node binaries. This helps significantly reduce the size of my Docker image (few MBs vs. ~700MB). This approach previously worked well with sharp, even in Lambda environments.

However, I've noticed some changes that seem to interfere with this setup.

I’ve read through the documentation on pnpm and webpack. However, when using:

externals: {
  'sharp': 'commonjs sharp'
}

...this assumes the library is discoverable at runtime, typically via node_modules. But in my setup, after bundling, that’s not the case.

From what I understand, this limitation stems from the way prebuilt binaries are resolved:

const paths = [
  `../src/build/Release/sharp-${runtimePlatform}.node`,
  '../src/build/Release/sharp-wasm32.node',
  `@img/sharp-${runtimePlatform}/sharp.node`,
  '@img/sharp-wasm32/sharp.node'
];

let sharp;
const errors = [];
for (const path of paths) {
  try {
    sharp = require(path);

This dynamic resolution makes it difficult for bundlers to include the relevant .node binaries, as these paths aren’t statically analyzable.

A possible workaround would be to hardcode the potential paths to make them discoverable during bundling, for example:

let sharp = undefined;
try { sharp = require(`../src/build/Release/sharp-linux-x64.node`,); } catch {...}
if (!sharp) try { sharp = require(`../src/build/Release/sharp-linux-arm.node`,); } catch {...}
if (!sharp) try { sharp = require(`../src/build/Release/sharp-linux-arm64.node`,); } catch {...}
if (!sharp) try { sharp = require(`@img/sharp-linux-x64/sharp.node`,); } catch {...}
[...]

Yes, it's tedious and not very elegant, but I haven’t found a better solution so far. I think it's acceptable to start with an initial list and expand it as needed.

Additionally, we could fall back to the current dynamic require logic at the end, preserving backward compatibility.

Does this approach make sense? Am I missing a better alternative?

Thanks again for the great work on sharp!

[lovell]

The relative paths between shared library binaries stored within node_modules are important, so even with the suggested changes above, bundling all the .node files together is not guaranteed to work.

What we could do is partly implement this by require-ing the @img/sharp-wasm32 WebAssembly package using a non-dynamic location, as that is a good fallback for almost all possible environments, plus should support whatever file system layout a bundler can throw at it.

[Bessonov]

@lovell

The relative paths between shared library binaries stored within node_modules are important, so even with the suggested changes above, bundling all the .node files together is not guaranteed to work.

Thanks! So you mean that the binaries communicate directly with each other and expect a relative - but exact - location? That's unexpected. I also see that libvips is included in the same way. Where can I learn more about this issue? I think I can configure Webpack to store the binaries at a specific path.

What we could do is partly implement this by require-ing the @img/sharp-wasm32 WebAssembly package using a non-dynamic location, as that is a good fallback for almost all possible environments, plus should support whatever file system layout a bundler can throw at it.

I'm fine with that approach. Does it have any drawbacks on Node.js, like performance degradation?

[Bessonov]

I have deployed my state to a test environment so it can be tested end to end. It seems to be working - at least for my use case with version 0.33.5. To replace the dynamic require with a hardcoded value, I'm using the following loader:

	module: {
		rules: [
			{
				test: /sharp\.js$/,
				loader: 'string-replace-loader',
				options: {
					search: 'sharp = require(path)',
					replace: "sharp = require('@img/sharp-linux-x64/sharp.node')",
				},
			},

[lovell]

Are you able to apply and test the following diff to see if that works for the scenario you (and hopefully others) are using? Thank you.

--- a/lib/sharp.js
+++ b/lib/sharp.js
@@ -13,8 +13,7 @@ const runtimePlatform = runtimePlatformArch();
 const paths = [
   `../src/build/Release/sharp-${runtimePlatform}.node`,
   '../src/build/Release/sharp-wasm32.node',
-  `@img/sharp-${runtimePlatform}/sharp.node`,
-  '@img/sharp-wasm32/sharp.node'
+  `@img/sharp-${runtimePlatform}/sharp.node`
 ];
 
 let path, sharp;
@@ -29,6 +28,15 @@ for (path of paths) {
   }
 }
 
+/* istanbul ignore next */
+if (!sharp) {
+  try {
+    sharp = require('@img/sharp-wasm32/sharp.node');
+  } catch (err) {
+    errors.push(err);
+  }
+}
+
 /* istanbul ignore next */
 if (sharp && path.startsWith('@img/sharp-linux-x64') && !sharp._isUsingX64V2()) {
   const err = new Error('Prebuilt binaries for linux-x64 require v2 microarchitecture');

[lovell]

@Bessonov Were you able to test the suggested change above?

[Bessonov]

@lovell Oh, I'm very sorry - it seems I missed the notification. I hope I can try it tomorrow.

[Bessonov]

@lovell It doesn't work out of the box.

First, I get the following error:

main-backend-1  | file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/sharp.js:131
main-backend-1  |   throw new Error(help.join('\n'));
main-backend-1  | ^
main-backend-1  | Error: Could not load the "sharp" module using the linux-x64 runtime
main-backend-1  | ENOENT: ENOENT: no such file or directory, open '/home/dev/app/packages/main-backend/dist/sharp-wasm32.node.wasm'

This happens because sharp-wasm32.node.js uses a plain path:

function findWasmBinary(){return locateFile("sharp-wasm32.node.wasm")}

If I copy the file manually, then I get this error:

main-backend-1  | file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/sharp.js:131
main-backend-1  |   throw new Error(help.join('\n'));
main-backend-1  | ^
main-backend-1  | Error: Could not load the "sharp" module using the linux-x64 runtime
main-backend-1  | Unsupported CPU: Prebuilt binaries for linux-x64 require v2 microarchitecture
[...]

This is due to an outdated path value and the following check:

if (sharp && path.startsWith('@img/sharp-linux-x64') && !sharp._isUsingX64V2()) {

After fixing that, I'm facing the following issue:

const logo = sharp(await uploadedFile.bytes()) // takes ~1ms, same as with sharp-linux-x64
const metadata = await logo.metadata() // not finished after >10 minutes; ~2ms with sharp-linux-x64
await logo.resize(...).toFile(...) // not finished after >10 minutes; ~24ms with sharp-linux-x64

There's nothing unusual in top or similar tools - no sign of high CPU or memory usage ¯\(ツ)/¯

Any ideas?

[Bessonov]

I have deployed my state to a test environment so it can be tested end to end. It seems to be working - at least for my use case with version 0.33.5. To replace the dynamic require with a hardcoded value, I'm using the following loader:

BTW, the workaround doesn't work with 0.34.2:

main-backend-1  | file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/utility.js:14
main-backend-1  | const libvipsVersion = sharp.libvipsVersion();
main-backend-1  | ^
main-backend-1  | TypeError: sharp.libvipsVersion is not a function
main-backend-1  |     at Object.../../.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/utility.js (file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/utility.js:14:1)
main-backend-1  |     at __webpack_require__ (file:///home/dev/app/packages/main-backend/dist/file:/webpack/bootstrap:19:1)
main-backend-1  |     at Object.../../.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/index.js (file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/[email protected]/node_modules/sharp/lib/index.js:14:1)
main-backend-1  |     at __webpack_require__ (file:///home/dev/app/packages/main-backend/dist/file:/webpack/bootstrap:19:1)
[...]

[lovell]

Thanks for the research.

First, I get the following error... This happens because sharp-wasm32.node.js uses a plain path:

This looks like it might relate to https://github.com/emscripten-core/emscripten/issues/22140 and suggests we need to try the emscripten EXPORT_ES6 flag, although there's a follow-up issue at https://github.com/emscripten-core/emscripten/issues/22508 that remains open.

[Bessonov]

I haven't tried it yet, but I think I can work around it. However, I have no idea why it hangs...

[Rush]

Also experiencing this problem and the workarounds above don't help.

[Abraham-Flutterwave]

Getting this today!

[Rush]

We're using this workaround:

        {
          test: /sharp\/lib\/sharp\.js$/,
          loader: 'string-replace-loader',
          options: {
            search: 'sharp = require(path)',
            replace:  existsSync(resolveModuleDir('sharp', `./src/build/Release/sharp-${process.platform}-${process.arch}.node`))
              ? `sharp = require('../src/build/Release/sharp-${process.platform}-${process.arch}.node')` :
                `sharp = require('@img/sharp-${process.platform}-${process.arch}/sharp.node')`,
          },
        },

[Bessonov]

@Rush @Abraham-Flutterwave Don't forget to vote on this issue. I'm not expecting it to make a big difference, but not voting definitely won't help.

[lovell]

The existing advice in the documentation to use Webpack's externals configuration remains.

https://sharp.pixelplumbing.com/install/#webpack

Please remember that Webpack is a code bundler and not a code packager. If you're packaging code e.g. for use on another machine, you'll need to ensure that everything required at runtime is part of that package after any bundling takes place. For example this might additionally include the contents of node_modules/sharp/** and node_modules/@img/**.

[Bessonov]

@lovell

Probably, it's something to define, but at least my use case isn't packaging in the sense of an "exe", but rather having all dependencies bundled in the right place without "trash" - which is the definition of a bundler.

The second part, about having the right version of dependencies at runtime, is correct but less relevant for me, because:

1. I’m using Docker everywhere, so only the kernel version is relevant.
2. I’m ensuring that my setup is consistent across all environments.

Taking into account that I'm not the only one, we can assume there's a demand for allowing Sharp to work with bundlers.

I've been using Sharp since 2016, and until the changes, it worked very well with Webpack. There are many other modules with binaries that work perfectly fine with Webpack, so it would be a real pity if we couldn't just use Sharp without a ton of workarounds.

Is there something we can do to help make it happen?

[Rush]

@Rush @Abraham-Flutterwave Don't forget to vote on this issue. I'm not expecting it to make a big difference, but not voting definitely won't help.

I am happy with my workaround. :-)

[Rush]

The existing advice in the documentation to use Webpack's externals configuration remains.

https://sharp.pixelplumbing.com/install/#webpack

Please remember that Webpack is a code bundler and not a code packager. If you're packaging code e.g. for use on another machine, you'll need to ensure that everything required at runtime is part of that package after any bundling takes place. For example this might additionally include the contents of node_modules/sharp/** and node_modules/@img/**.

Bundling is extremely useful for a) minimizing docker size b) making sure we can bundle with a proper lockfile and then have only fully bundled code in production, with no external node_modules.