Processed images are published without invoking .Permalink, .RelPermalink or .Publish

[dgerhardt]

According to the docs, assets in the assetDir directory are only published by invoking .Permalink, .RelPermalink, or .Publish. This is true for the source images but once image processing methods are applied, the result is automatically published. This can have a significant impact on the size of build artifacts, especially when multiple operations are performed on images.

For my use case, I am resizing two images, merge them with the overlay filter, convert them to webp and then generate a fingerprinted permalink. This leads to a huge number of published image variants which are never used.

Update: The issue seems to only occur once the cache exists. When cacheDir is deleted before the build, the issue can no longer be reproduced.

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.102.3-b76146b129d7caa52417f8e914fc5b9271bf56fc+extended linux/amd64 BuildDate=2022-09-01T10:16:19Z VendorInfo=gohugoio

Does this issue reproduce with the latest release?

~~Did not check against v0.102.x because klakegg's Docker images are not available yet but there are no related changes documented for this version.~~ Yes, tested with v0.102.3.

[jmooring]

@dgerhardt I am unable to reproduce this:

template

{{ $i := resources.Get "a.jpg" }}
{{ $i = $i.Resize "300x webp" }}

commands

rm -rf resources/ public/
hugo
tree public resources

result

public/
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        └── a_hu4cf93b69df91308fdf11e4cd6cd70889_17959_300x0_resize_q75_h2_box.webp

The resource is cached, but not published (it's not in the public directory).

I'm probably missing something. Can you provide a more detailed example?

[dgerhardt]

Can you provide a more detailed example?

@jmooring Sure. You can try the following example to reproduce the issue:

{{ $image := resources.GetRemote "https://upload.wikimedia.org/wikipedia/commons/thumb/4/41/Shtandart_%28ship%2C_1999%29%2C_S%C3%A8te_cf02.jpg/500px-Shtandart_%28ship%2C_1999%29%2C_S%C3%A8te_cf02.jpg" }}
{{ $image = $image.Resize "400x" }}
{{ $image = $image.Resize "300x" }}
{{ $image = $image.Resize "200x" }}
{{ $image = $image.Resize "100x" }}
{{ $image = $image.Crop "50x50" }}
{{ $image = $image.Filter (images.GaussianBlur 6) }}
{{ ($image | fingerprint).Permalink }}

This will result in 7 published image files:

$ ls -1p public/
categories/
_hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
index.html
index.xml
sitemap.xml
tags/

[jmooring]

@dgerhardt Using your example I get this:

public/
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
        └── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg

Here's a test repo to try:

git clone --single-branch -b hugo-github-issue-10255 https://github.com/jmooring/hugo-testing hugo-github-issue-10255
cd hugo-github-issue-10255
hugo
tree public resources

The template code is in layouts/_default/home.html. Running hugo v0.102.3.

[dgerhardt]

@jmooring Thank you for further looking into this. I've now switched from the Docker images to the the official v102.3 binary to make sure the version wasn't the cause of the issue. I was able to reproduce the issue with your repository but it only happens once the cache exists.

Fresh start without cache:

$ rm -rf public resources
$ hugo
$ tree -F public resources
public/
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
        └── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg

Only delete public this time - keep the cache:

$ rm -rf public
$ hugo
$ tree -F public resources
public/
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
        └── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg

[jmooring]

Confirmed. MRE:

git clone --single-branch -b hugo-github-issue-10255 https://github.com/jmooring/hugo-testing hugo-github-issue-10255
cd hugo-github-issue-10255
hugo && tree public resources # Results are expected
hugo && tree public resources # Results are NOT expected

[bep]

I think I have observed this myself, and I'm also pretty sure I have this fixed in a branch that has taken me a bit longer to get over the finish line than I planned. I have been side stepped a little lately, but I plan to pick up this particular ball on Monday.

[dgerhardt]

@bep Since you mentioned that you might already have a fix: Did you have the chance to have another look at it?

[xlsigned]

Reproduced on

• hugo v0.110.0+extended linux/amd64 (ArchLinux)
• hugo v0.111.1-39a4a3cf676533859217805c36181c7a3dfa321c+extended windows/amd64 (Windows 10)
• hugo v0.111.2-4164f8fef9d71f50ef3962897e319ab6219a1dad+extended windows/amd64 (Windows 10)

This also happens with image resources from a headless bundle, e. g. a sub-folder of a top-level section, with publishResources = false.

It's even worse:

• When I use a custom figure shortcode which does some image processing before generating the <img src="{{ $image.RelPermalink}}" />, I get all those intermediate images published and not only the final result.
• When I use the builtin figure shortcode to reference the original image, it's missing in the generated output (as soon as I use publishResources = false).
• When I use Markdown syntax to reference the original image, i. e. ![Image](./headless/image.png), it's missing in the generated output (as soon as I use publishResources = false)
• When I do not use publishResources = false, everything from the headless bundle folder gets published, no matter if referenced or not. Even a Makefile and a graphviz .gv file I use to generate the original image .png file.

This is clearly not what I was expecting. The reason why I set publishResources = false on the headless bundle was that I wanted exactly the referenced resources to be published. Neither any intermediate results nor sources I use to generate the resources.

https://gohugo.io/content-management/build-options/#publishresources

Dunno, if it matters, but I have a dual-language setup, i. e. the headless bundle has (and needs) an index.md and an index.de.md file to be usable (be referenced) from pages of both languages.

I actually/initially thought a headless bundle would work like this (only referenced resources are published) without explicitly setting publishResources = false. I was also very surprised to see that I needed index.LANG.md files to access the headless bundle resources from other than the default language.

I'm currently trying to figure out under which conditions a page resource is (supposed to be) published:

• https://github.com/gohugoio/hugo/search?q=publishresources&type=code
• https://github.com/gohugoio/hugo/search?q=lazypublish&type=code
• https://github.com/gohugoio/hugo/search?q=publishonce&type=code

[hrishikesh-k]

I noticed another situation in which this could cause issues. I had created a custom image.html partial that was processing any image I pass to it. I added that image to ./layouts/_default/baseof.html as I needed it on every page. Since I'm also using resources.Copy, I ended up with 2 images (I currently have just 2 pages, didn't test with more pages). The first image was the required image (the one that I produced with resources.Copy) and the second one was the one copies from cache.

This was happening, even if I delete my resources folder, probably because I am rendering it twice. If I add it just to index.html by removing it from baseof.html, and delete the resources directory, I just get the image once.

I solved this by using partialCached, so this might not be as much of an issue, but I thought I'd let you know.

[shreve]

I solved this by using partialCached

Thanks, that worked for me for a related issue.

I was using a resized image in my _default/baseof and the final file in public/ was ending up empty. Extracting to a partial and using partialCache resolved that.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.