PSNR metric returns `all: 1` resulting in incorrect results

[vishaalagartha]

When I run compare -verbose -metric psnr test.png test.png output.png (comparison on the same file)` this is the output I get:

test.png PNG 3456x2234 3456x2234+0+0 8-bit TrueColorAlpha sRGB 1.03965MiB 0.060u 0:00.063
test.png PNG 3456x2234 3456x2234+0+0 8-bit TrueColorAlpha sRGB 1.03965MiB 0.050u 0:00.050
Image: test.png
  Channel distortion: PSNR
    red: 1
    green: 1
    blue: 1
    alpha: 0
    all: 1
writing raw profile: type=icc, length=4064
writing raw profile: type=xmp, length=450
test.png=>output.png PNG 3456x2234 8-bit sRGB 1.03965MiB 0.640u 0:00.417

It appears that imagemagick can also output 1 if the files are identical? Hence, shouldn't this line (https://github.com/bgeron/diff-pdf-visually/blob/16513817e5c116507b3e08e4cdca72b1d53e7463/diff_pdf_visually/diff.py#L79C1-L80C1) all_num = INFINITY if (all_str == "0" or all_str == "1.#INF") else float(all_str) be changed to all_num = INFINITY if (all_str == "0" or all_str == "1.#INF" or all_str == "1") else float(all_str) ?

Thanks in advance! I manually changed this in my site_packages and things to be working fine now.

ImageMagick version: 7.1.2-0 Q16-HDRI aarch64 23234 diff_pdf_visually version: 1.3.1

[xljones]

Had the same issue. Setting the threshold explicitly to 1.00 in your call to pdf_similar() should also resolve the issue, as the DEFAULT_THRESHOLD of 100 is of course unreachable! This should allow an ImageMagick response of Inf or 1 to be valid as a perfect match.

[xljones]

@bgeron would you be open to a PR to fix this? Will take a look later this week 🙌

[bgeron]

Hey @xljones, for sure! Sorry I let this ticket slip when my life was being busy. I should be able to respond pretty quickly, in the unlikely case it does slip off my radar please do nag me about it.

Thanks for looking into this!

[Younith]

Had the same issue, the solution proposed above fixed it!

Although, switching to a different metric also worked, so it might be worth exploring an option to specify the metric instead of just PSNR.

[bgeron]

Looking into this, tests are failing but this seems related to ImageMagick instead. Expect a release with a fix tomorrow evening.

[bgeron]

Looked more into this, and the problem is unfortunately much bigger. I will try to get to this latest at the weekend. edit: making progress but didn't end up having enough time.

@vishaalagartha, with ImageMagick 7.1.2 (August 2025) you are running a new version. I assume @Younith is too.

Since ImageMagick 7.1.44 (March 2025), the reported PSNR values are completely opposite.

• Old ImageMagick: same images = PSNR infinity, bit different = PSNR 30dB, very different = PSNR 5db. More different is smaller PSNR.
• Modern ImageMagick: same images = PSNR 0, bit different = PSNR 0,000000001, very different = PSNR 0,1. More different is larger PSNR.

(I'm just making up some numbers for the example)

You can imagine the threshold computation is completely screwed up now. In the past, we reported "different" when PSNR < threshold. Now we should report "different" when PSNR > threshold.

The problem is even bigger: my local ImageMagick 7.1.1-47 reports PSNR=1 when comparing a black and a white image. When comparing the same image twice, I get PSNR=0. It would not really be cool if my tool says that a fully black and a fully white PDF image are the same. So I don't really know what's going on with ImageMagick 7.1.2-0. Maybe they inverted the logic a second time? Any hints will be most helpful.

@vishaalagartha Is your test.png completely transparent by any chance? If you have time to investigate, could you compare -verbose -metric PSNR white.png white.png diff.png and compare -verbose -metric PSNR white.png black.png diff.png (images attached) and report the results? I'm getting all: 0 when comparing black/black and white/white, and all: 1 when comparing black/white and white/black.

@xljones , do you also get all: 1 like @vishaalagartha when comparing an image with itself, or do you get all: 0 like me? And which version of ImageMagick do you have (compare -version)?

black.png white.png

To be honest this needs a bunch of changes in multiple places. I have an early draft but it looks like a lot of work is needed to catch up with the last 9 months of ImageMagick.

Thanks for any info you can help with!

[bgeron]

I did some research, and different versions of ImageMagick seem to have these scales:

• 7.1.1-0…7.1.1-45: 0 = same → ∞ (-0) = completely different
• 7.1.1-46: broken: 0 can mean the same or completely different
• 7.1.1-47: 0 = same → 1 = completely different
• 7.1.2-0…7.1.2-8: 1 = same → 0 = completely different

I'm making a new version that detects the version of ImageMagick installed, and behaves right for that version.

[bgeron]

update: I calculated some metrics for examples between v7.1.1-45, v7.1.1-47, and v7.1.2-8, and I figured out some formula to make the metrics comparable, but it also seems that it's not simply a formula but an algorithm change. For instance, it seems like v7.1.1-45 doesn't care as much about layout shifts as higher versions. However, we can still bring the numbers on a similar scale to make them somewhat comparable.

[bgeron]

I hope I fixed the issue in new prerelease version 2.0.0.a1. Try it with:

uvx [email protected]

or using your package manager of choice.

This version detects the installed version of ImageMagick, and converts all metrics to the modern range: from 0 to 1, higher is more similar.

@vishaalagartha @xljones @Younith If you have time, could you try some of your use cases with this version, see if things are working or if you see anything surprising? I would like to release this as a new major version, but there's a lot of new code and I'd like to avoid accidentally breaking anything for others.