out of memory with too many/big images

[abuu-u]

when i run next-optimized-images with a lot of images on a machine with 16gb ram, it crashes and i am getting this error:

ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL  Command was killed with SIGKILL (Forced termination): next-export-op

i think the issue is loading same images for different sizes multiple times. so i changed it to load image one time for one src and use it for creating different size images and it worked without crashes.

[abuu-u]

and tested this with less images and got slightly faster builds

without fix:

rm -rf out .next node_modules/.cache/next-export-optimize-images/ ; pnpm build ; time -v pnpm next-export-optimize-images
...
Successful optimization!
        Command being timed: "pnpm next-export-optimize-images"
        User time (seconds): 373.81
        System time (seconds): 35.86
        Percent of CPU this job got: 386%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:45.86
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 13222108
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 144121
        Minor (reclaiming a frame) page faults: 5239105
        Voluntary context switches: 1761048
        Involuntary context switches: 158345
        Swaps: 0
        File system inputs: 16854392
        File system outputs: 510712
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

and with fix:

rm -rf out .next node_modules/.cache/next-export-optimize-images/ ; pnpm build ; time -v pnpm next-export-optimize-images
...
Successful optimization!
        Command being timed: "pnpm next-export-optimize-images"
        User time (seconds): 356.88
        System time (seconds): 20.95
        Percent of CPU this job got: 433%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:27.09
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2348060
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 2
        Minor (reclaiming a frame) page faults: 396526
        Voluntary context switches: 181195
        Involuntary context switches: 34491
        Swaps: 0
        File system inputs: 1352
        File system outputs: 510712
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[abuu-u]

this is roughly what i changed (i edited complied code directly from node_modules):

@@ -44,6 +44,7 @@ type GetOptimizeResultProps = {
 type GetOptimizeResult = (getOptimizeResultProps: GetOptimizeResultProps) => Promise<void>

 export const getOptimizeResult: GetOptimizeResult = async ({
+  imageBuffer,
   destDir,
   noCache,
   cacheImages,
@@ -68,8 +69,6 @@ export const getOptimizeResult: GetOptimizeResult = async ({
       const outputPath = path.join(cacheDir, output)
       await fs.ensureFile(outputPath)

-      const imageBuffer = await fs.readFile(originalFilePath)
-
       // Cache process
       if (!noCache) {
         const cacheImagesFindIndex = cacheImages.findIndex((cacheImage) => cacheImage.output === output)
@@ -278,26 +277,44 @@ export const optimizeImages = async ({
   const errorMeasurement = () => (measuredError += 1)
   const pushInvalidFormatAssets = (asset: string) => invalidFormatAssets.add(asset)

-  for (const item of manifest) {
-    const originalFilePath = path.join(destDir, item.src)
-
-    promises.push(
-      getOptimizeResult({
-        destDir,
-        noCache,
-        cacheImages,
-        cacheDir: defaultCacheDir,
-        cacheMeasurement,
-        nonCacheMeasurement,
-        errorMeasurement,
-        pushInvalidFormatAssets,
-        cliProgressBarIncrement: terse ? () => undefined : cliProgressBarIncrement,
-        originalFilePath,
-        quality: config.quality ?? 75,
-        sharpOptions: config.sharpOptions ?? {},
-        ...item,
-      })
-    )
+  manifest = manifest.reduce((prev, curr) => {
+    if (!prev[curr.src]) {
+      prev[curr.src] = [curr];
+    } else {
+      prev[curr.src].push(curr);
+    }
+
+    return prev
+  }, {})
+
+  for (const [key, value] of Object.entries(manifest)) {
+    const imageBuffer = await fs.readFile(path.join(destDir, key));
+
+    promises.push((async () => {
+      const promises2 = [];
+
+      for (const item of value) {
+        const originalFilePath = path.join(destDir, item.src);
+
+        promises2.push(getOptimizeResult({
+          imageBuffer,
+          noCache,
+          cacheImages,
+          cacheDir: defaultCacheDir,
+          cacheMeasurement,
+          nonCacheMeasurement,
+          errorMeasurement,
+          pushInvalidFormatAssets,
+          cliProgressBarIncrement: terse ? () => undefined : cliProgressBarIncrement,
+          originalFilePath,
+          quality: config.quality ?? 75,
+          sharpOptions: config.sharpOptions ?? {},
+          ...item,
+        }))
+      }
+
+      await Promise.all(promises2);
+    })())
   }

   await Promise.all(promises)

[dc7290]

Looks good! I'll give it a try.

[github-actions[bot]]

🎉 This issue is included in version v4.3.0.