Optimizing the process of converting a GIF with numerous frames to prevent memory issues.

[oluSammy]

What are you trying to achieve?

conversion of gif to webp. sample image - https://res.cloudinary.com/olumorinsamuel/image/upload/v1706112890/sample-gif_aih0ge.gif

With 1007 frames in this GIF, I encounter a memory issue when attempting to convert it to the WebP format using a lambda function. It seems that the lambda function exhausts its memory quickly, likely due to Sharp attempting to load all frames simultaneously. I'm seeking a solution to perform the conversion in batches, mitigating memory exhaustion. It's worth noting that I've already allocated the maximum memory available for a lambda function, which is 10GB.

Error after 234ms

{
  "errorType": "Runtime.ExitError",
  "errorMessage": "RequestId: a2d7fb8e-1f24-410a-b035-09b0d744a157 Error: Runtime exited with error: signal: killed",
  "message": "Error: vips_tracked: out of memory --- size == 10375MB\n",
}

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

Here is the code

  const options: sharp.SharpOptions = {
    limitInputPixels: false,
    failOnError: false,
   animated: true
  };


sharp.cache({ memory: 0, files: 0, items: 0 });
const input = fs.createReadStream("input-path");
const output = fs.createWriteStream(`output-path`);

input.on("error", (err) => {
  reject(err);
});

input
  .pipe(
    sharp(options)
      .resize(size)
      .webp({ quality:  100 })
  )
  .on("error", (err) => {
    reject(err);
  })
  .pipe(output)
  .on("error", (err) => {
    reject(err);
  })

[lovell]

const input = fs.createReadStream("input-path"); const output = fs.createWriteStream(`output-path`);

If the input and output are on the filesystem then you can provide the paths direct to sharp. The use of Streams will increase memory pressure as they involve lots of small de/allocations.

I tried your sample-gif_aih0ge.gif image locally, using filesystem input and output, and it consumed ~120 MB at peak. When adding in resizing to 200x200 it consumed ~40 MB at peak.

(This particular image would probably be better off in a vector format e.g. SVG or perhaps even text and a bit of CSS/JavaScript to style/animate it.)

[lovell]

I hope this information helped. Please feel free to re-open with more details if further assistance is required.