img-dim is slow and not cached

[theor]

Hello,

I realized that the img processing is executed multiple times per image - typically, if I add an image to a blog post, it will get processed 3 times (the blog post page itself, the /posts list page, the / list page) + once per tag . I have a site icon processed 22 times per build, for a total of 1721ms according to the 11ty benchmark log.

Caching would be great ; I see a solution in 2 steps :

• keep an in-memory, transient cache indexed by path: should be easy to do. fill the cache in img-dim, invalidate it when the image changes. pay the cost once per image when starting watch mode, still way better
• then get to a persistent cache. we could write an image-hashes.json file with the previous hash and generated file path, check it once during watch startup and compare the stored hashes against the actual hashes to invalidate or not the cache entries.

Thoughts ?

Also: it looks like gif2mp4 (which is even slower) is not running if an mp4 file already exists in the source folder (see https://github.com/google/eleventy-high-performance-blog/blob/676789e95149e0c92aee4bb4b89cabc6680db1c0/_11ty/video-gif.js#L32 ). is that intended to act as a basic cache ? in that case it should be if (await exists(join("_site", dest))) { instead of if (await exists(dest)) {

[cramforce]

Did you see which part of img-dim is slow?

I'm definitely supportive of caching it. I just have never seen it being slow. You definitely found a bug for the GIF special case, but otherwise the image processing is the slow part and obviously that is being done only once per build.

[theor]

await sizeOf("_site/" + src); is slow and done every time (that's where the time is spent the next 21 times in my example) srcset doing all the actual resizing is slow too, but done only once

[cramforce]

OK. sizeOf is an O(1) operation but I can see it getting a factor for N images :)

I'm definitely supportive of caching it.

[cozarkd]

I'm having serious problems with the image processing scripts. My project had an average of 1-2min building time, now it's +20min. So I can't use that in netlify or cloudflare pages. Probably need to do it locally in a separate script and store the files.

[indcoder]

Agree with @cozarkd . It has dramatically increased the build time and it is taking up so much memory that we cannot now use Netlify for building [so CI/CD is busted].

[cramforce]

Did you try running .persistimages.sh after a local build? It should eliminate all slowness.

[cozarkd]

Did you try running .persistimages.sh after a local build? It should eliminate all slowness.

oh, I thought that file was for remote images, makes sense now. In my use case I had to implement some features but not the full template. Problem solved, thanks