"Performance" section on web site

[akx]

Hi there!

Props for the nice library. I'm curious about the performance numbers on the web site. It says

Photon outperforms even the fastest of libraries, both natively and on the web. Photon has greatly outperformed ImageMagick and the Python Imaging Library, as shown in the graph to the right

but there's no note on what exactly was benchmarked in each case, or how. Could that information be added to the site? Right now it just says Photon does something in 2 seconds, ImageMagick does something in 22, and PIL (btw, did you use PIL or Pillow?) does something in 40. 😅

[silvia-odwyer]

Hi Aarni! Thanks for mentioning this! 😄 I agree that more specific benchmarks should be included on the site and beyond, it was on my list but I'm going to put that higher up in priority so that I'll have this completed soon.

I'll make sure to add some specific benchmarks to the site, and maybe add a Wiki page here which includes further details and benchmark results. I also have a benchmarking library integrated currently, which runs benchmarks for gaussian blur, so that those who wish to benchmark on their own machines (with images of their choice) can do so when they execute cargo bench, plus view outputted graphs and so forth. I may also upload these graphs to the site.

Thanks again for mentioning this, much appreciated

[zimond]

I kindly advise that this section be removed, along with the Photon outperforms even the fastest of libraries, including ImageMagick in readme. Here are my thoughts:

1. ImageMagick is not well accepted as the fastest image library
2. A library has to be really neatly benched to be declared as one of the "fastest"
3. The known fastest (maybe) library based on CPU image processing is pillow-simd. And I believe photon cannot beat its performance in a short time 3.1. pillow-simd heavily uses SIMD instructions, which is not implemented by any rust image processing lib now 3.2. pillow-simd uses libpng and libjpegturbo, both uses SIMD to do encoding/decoding and image is not using them.
4. GPU-based image processing could easily beat CPU-based processing in performance
5. photon as an image processing library, should not rely on image codec performance, which is not his job as intended. (fast png codec doesn't mean the image processing is fast).

[zimond]

By the way, if benchmark is really that important, there's a famous and interesting read about pillow-simd (and how hard to beat it) here:

https://gist.github.com/brian-armstrong-discord/1eb10046edb91167b7187513cc306d65

[jcupitt]

Hello, this looks like a nice project. But I wonder about the benchmark -- running:

vips invert sample.png x.png

Spends ~75% of CPU in libz, and more than 90% in libpng (according to callgrind). I'm not sure you're really measuring the speed of either project.

FWIW, my personal preference is for a defined task rather than a set of microbenchmarks. The libvips benchmark is 1) load an image 2) crop 3) resize 4) sharpen 5) save. It's not very sophisticated, but does stress load, save, manipulate, filter and resize, which are (arguably) the most most important basic operations a library needs to provide. It uses uncompressed TIFF as the image format, since libtiff is extremely fast (not much more than repeated read() calls).

[silvia-odwyer]

Hi John, that's a good point about the nature of the benchmark conducted. I'll run more benchmarks consisting of further operations, such as cropping and resizing also, which will provide a better insight into the performance of the library. I'll make sure to update the Wiki benchmarks page as soon as I complete these benchmarks.

[akx]

The benchmarks page at https://github.com/silvia-odwyer/photon/wiki/Benchmarks still doesn't show any methodology for the Python Imaging Library results 😞

[jcupitt]

The first test, for example, will be spending 90% of the run time inside libpng, so the important factors in the observed speed will be:

• what version of libpng was used (current libvips is using libspng for decode, for example)
• how libpng was built (optimization flags etc.)
• the version of zlib that libpng was built against (zlib-ng can give a 30% speed improvement)
• the parameters that libpng was passed (compression is the critical one -- libvips is using 6, the default, but setting 0 will give a x4 speedup)

Perhaps you could show the sizes of the output files? It would be simple way to demonstrate that the two systems are doing approximately the same task.

Demo:

$ time vips copy nina.jpg x0.png[compression=0]
real	0m1.208s
$ time vips copy nina.jpg x6.png
real	0m5.324s
09:01 $ ls -l x*.png
-rw-r--r--@ 1 john  staff  73279068  1 Sep 09:03 x0.png
-rw-r--r--@ 1 john  staff  11855722  1 Sep 09:01 x6.png

[jcupitt]

I tried adding a PNG invert task to the benchmark (the one in the repo is JPG):

fn invert_image_png() {
    // Open the image (a PhotonImage is returned)
    let mut img = open_image("examples/input_images/fruit.png");

    // Invert the image
    photon_rs::channels::invert(&mut img);

    let output_img_path = "output-invert.png";

    // Write to filesystem
    save_image(img, output_img_path);
}

fruit.png is the image of fruit slices linked from the benchmark page. I see:

Benchmarking invert_image_png: Collecting 10 samples in estimated 30.202 s (110                                                                                 invert_image_png        time:   [272.85 ms 273.39 ms 273.85 ms]

The file is:

$ ls -l output-invert.png 
-rw-rw-r-- 1 john john 3094638 Sep  1 10:16 output-invert.png

If I try with libvips 8.10 I see:

$ time vips invert examples/input_images/fruit.png output-vips.png
real	0m0.956s
user	0m1.018s
sys	0m0.022s
$ ls -l output-vips.png 
-rw-r--r-- 1 john john 2537537 Sep  1 10:22 output-vips.png

So libvips does seem to be using a higher setting for compression (2.5MB vs. 3MB). If I try with compression 1, I see:

$ time vips invert examples/input_images/fruit.png output-vips.png[compression=1]
real	0m0.313s
user	0m0.359s
sys	0m0.034s
$ ls -l output-vips.png 
-rw-r--r-- 1 john john 2916339 Sep  1 10:39 output-vips.png

So about the same speed as photon, but a smaller file size. There must be another setting that's different between the two libraries.

[kleisauke]

I tried to add the WebAssembly version of Photon to wasm-vips' performance benchmarks, see commit https://github.com/kleisauke/wasm-vips/commit/f38199f8f7301dad18ee40ad9c506f7672dd906b. On that benchmark, wasm-vips is 2 times faster than Photon.

As John noticed, it's currently impossible to specify the zlib compression or row filter(s) when saving to PNG. It seems that these 2 settings default to the libvips setting [compression=1,filter=sub] in image-rs. https://github.com/image-rs/image/blob/v0.23.12/src/codecs/png.rs#L507-L508

So, the PNG benchmarks on https://github.com/silvia-odwyer/photon/wiki/Benchmarks are a bit misleading, since libvips spends much more CPU effort on improving PNG compression.