Resizing and cropping on demand

[drewlr]

Hi,

This is more a request for advice than issue, if you can help!

The use case is a UI that allows to load images and zoom in / span across the image. Images can be in different formats (JPG, PNG, TIFF), that can be say from KBs to 1-2GB.

The approach we are following at the moment is to differentiate between big and small images.

For small images, we might not use pyvips at all since it seems that the overhead kills the performance gain.

For big images: we are loading first a low resolution image which we create beforehand. Then upon zooming, we are loading the relevant region that needs zooming (or rather, a slightly larger and high resolution region than needed, to minimise the number of calls made to libvips). For this, we are currently doing something like:

image = pyvips.Image.new_from_file(filename, access='sequential') image = image.crop() out = image.resize() data = out.write_to_buffer('.jpg', Q=quality)

Some questions:

0. Is this the most efficient way to approach the problem?
1. Reading through other issues (https://github.com/libvips/pyvips/issues/26) and libvips documentation, I'm thinking that something like the below should instead be quite faster?

im = pyvips.Image.thumbnail(filename, ) im = im.crop() data = im.write_to_buffer('.jpg', Q=quality)

2. Is it worth to have different approaches depending on the format of the image?
3. would something like direct access to the relevant region of the image be more efficient? But I could not find much documentation on this

Thanks a lot

[jcupitt]

Hello, thumbnail plus crop will be extremely slow. It's better to make a pipeline which generates the whole image, then ask for chunks of that.

Is this for a desktop program or a web page?

For a web page, I would make a small tile server in C, eg.:

https://github.com/jcupitt/tilesrv

Then make the interface in javascript.

If it's desktop, you need vips_sink_screen():

https://libvips.github.io/libvips/API/current/libvips-generate.html#vips-sink-screen

It lets you connect the output of a vips image processing pipeline to a window. It's all asynchronous, so screen pixels are generated in the background and painted to the screen as they are computed.

Here's a tiny demo program to show how to use it:

https://github.com/libvips/vipsdisp-tiny

You need to take the core of that and wrap it up as a widget that can display a VipsImage, then you can use pygobject / pyvips / etc. to write the GUI.

Here's a complete image viewer in C done in this way:

https://github.com/jcupitt/vipsdisp

Try it with a large image. Use + and - to zoom, cursor keys to pan, etc. It's not quite finished, unfortunately, I must get around to it.

Have you seen nip2? It is very similar, and can display large images quickly.

[jcupitt]

Oh, vipsdisp has other keys:

1, 2, 3, 4 ... pick a zoom level. ctrl + 1, 2, 3, 4 ... pick a zoom out level 0 ... zoom to fit shift + cursor ... move a page at a time ctrl + cursor ... move to image edges left mouse + drag ... pan +, - ... zoom in or out on the mouse i, o ... zoom in or out bottom sliders ... set display scale and offset (scale should be log, I must fix this)

It's supposed to be the window display widget for nip3, so most of the features are copied from that.

vipsdisp defines a widget called Imagedisplay (a subclass of DrawingArea) which you should be able to use from Python.

[jcupitt]

Is this for jupyter?

That would be tricky. I think you would have to output some javascript which implemented the viewer (perhaps with openseadragon), and then connect that to a tiny tile server that in turn was given the image to display.

I've no idea if jupyter lets you do that though -- it has a pretty limited display system (as far as I know).

[drewlr]

Thanks for the pointers @jcupitt !

It's for a webapp (to be ran either in browser or in an Electron.JS wrapper) that should be able to run locally on Mac/Linux/Windows. We have a React.JS frontend - we need to make the zoom and span experience fast, the specific use case is images annotation for Computer Vision (I guess you must be getting a lot of traffic due to Deep Learning these days! :) )

We can't really create tiles beforehand for all the pictures as that could eat up a lot of disk space.

Waiting ~2s per image when needing to zoom in is not ideal either I guess, but possibly that could work - it would come into play only for big images. We will do some tests on our end using your pointers, thanks!

[jcupitt]

The 2s is for openslide images (up to about 300,000 x 300,000 pixels), it'd be quicker for more normal files.

I made a node binding for libvips:

https://github.com/libvips/node-vips

You could probably write the tile server in that. I did it as an exercise and I don't know anything about node, so it might need a bit of fixing up. It's only ~2,000 lines, so it should be easy to work on.