Quit the saving process with pyvips.image.tiffsave

[yzhang525]

Hello,

I have been using Pyvips to process the whole slide images. I noticed that it usually took very long time to save pyramidal tiffs using pyvips.image.tiffsave (could be hours). My question is that, during this saving process, after some time if I decide not to keep waiting, is there a way that I can quit the tiffsave function rather than doing "Ctrl + C" in the terminal? I'd like to create a quit function.

Any help would be much appreciated!

[jcupitt]

Hello, there's some sample code here:

https://github.com/libvips/pyvips/blob/master/examples/progress.py

You can monitor save progress and kill computation.

[yzhang525]

Thank you for your response, John!

I just did some quick test with my code as follows, but it seems the added code didn't kill the process. Do you think what I did wrong or do you have any suggestions? Thank you!

regComplete.set_progress(True)

regComplete.tiffsave(os.path.join(out_path, f"Output.ome.btf"),
                                                                            compression='lzw', bigtiff=True, 
                                                                            pyramid=True, subifd=True,
                                                                            tile=True, depth='onetile',
                                                                            tile_width=512, tile_height=512)

def progress_print(name, progress):
    print('{}:'.format(name))
    print('   run = {}'.format(progress.run))
    print('   eta = {}'.format(progress.eta))
    print('   tpels = {}'.format(progress.tpels))
    print('   npels = {}'.format(progress.npels))
    print('   percent = {}'.format(progress.percent))


def eval_cb(image, progress):
    progress_print('eval', progress)
    # you can kill computation if necessary
    if progress.percent > 50:
        image.set_kill(True)

regComplete.signal_connect('eval', eval_cb)

[jcupitt]

Sorry, I don't know, you'd need to post a complete program I can test.

Did the example work for you? I see:

$ ./progress.py 
preeval:
   run = 0
   eta = 0
   tpels = 500
   npels = 0
   percent = 0
eval:
   run = 0
   eta = 0
   tpels = 500
   npels = 320
   percent = 64
eval:
   run = 0
   eta = 0
   tpels = 500
   npels = 384
   percent = 76
eval:
   run = 0
   eta = 0
   tpels = 500
   npels = 384
   percent = 76
eval:
   run = 0
   eta = 0
   tpels = 500
   npels = 400
   percent = 80
eval:
   run = 0
   eta = 0
   tpels = 500
   npels = 400
   percent = 80
posteval:
   run = 0
   eta = 0
   tpels = 500
   npels = 500
   percent = 100
Traceback (most recent call last):
  File "/home/john/GIT/pyvips/examples/./progress.py", line 36, in <module>
    image.avg()
  File "/home/john/.local/lib/python3.10/site-packages/pyvips/vimage.py", line 1347, in call_function
    return pyvips.Operation.call(name, self, *args, **kwargs)
  File "/home/john/.local/lib/python3.10/site-packages/pyvips/voperation.py", line 305, in call
    raise Error('unable to call {0}'.format(operation_name))
pyvips.error.Error: unable to call avg
  VipsImage: killed for image "temp-0"

[yzhang525]

Thank you, John! The entire code is pretty long. But the basic idea related to my question above is to read in a whole slide image and save it in a pyramidal structure using pyvips. In the above code, regComplete = pyvips.image.newfromfile[...] . All the information is restored in regComplete. Then my understanding is that once I call regComplete.tiffsave(...), then I don't have any way to kill the process. Is my understanding right?

[jcupitt]

You should be able to cancel. But you need to show me a complete program that fails to work before I can say what's wrong.

Here's another demo:

#!/usr/bin/python3

import sys
import pyvips


def progress_print(name, progress):
    print(f'signal {name}:'.format(name))
    print(f'   run = {progress.run} (seconds of run time)')
    print(f'   eta = {progress.eta} (estimated seconds left)')
    print(f'   tpels = {progress.tpels} (total number of pels)')
    print(f'   npels = {progress.npels} (number of pels computed so far)')
    print(f'   percent = {progress.percent} (percent complete)')


def preeval_cb(image, progress):
    progress_print('preeval', progress)


def eval_cb(image, progress):
    progress_print('eval', progress)

    # you can kill computation if necessary
    if progress.percent > 50:
        image.set_kill(True)


def posteval_cb(image, progress):
    progress_print('posteval', progress)


image = pyvips.Image.new_from_file(sys.argv[1], access="sequential")
image.set_progress(True)
image.signal_connect('preeval', preeval_cb)
image.signal_connect('eval', eval_cb)
image.signal_connect('posteval', posteval_cb)
image.write_to_file(sys.argv[2])

So it's just copying a file. I can run:

$ ./copy-with-cancel.py ~/pics/openslide/CMU-1.svs x.dz
signal preeval:
   run = 0 (seconds of run time)
   eta = 0 (estimated seconds left)
   tpels = 1514044000 (total number of pels)
   npels = 0 (number of pels computed so far)
   percent = 0 (percent complete)
signal eval:
   run = 0 (seconds of run time)
   eta = 0 (estimated seconds left)
   tpels = 1514044000 (total number of pels)
   npels = 311296 (number of pels computed so far)
   percent = 0 (percent complete)
...
signal eval:
   run = 15 (seconds of run time)
   eta = 14 (estimated seconds left)
   tpels = 1514044000 (total number of pels)
   npels = 772212736 (number of pels computed so far)
   percent = 51 (percent complete)
signal posteval:
   run = 15 (seconds of run time)
   eta = 0 (estimated seconds left)
   tpels = 1514044000 (total number of pels)
   npels = 1514044000 (number of pels computed so far)
   percent = 100 (percent complete)
Traceback (most recent call last):
  File "/home/john/try/./copy-with-cancel.py", line 37, in <module>
    image.write_to_file(sys.argv[2])
  File "/home/john/.local/lib/python3.10/site-packages/pyvips/vimage.py", line 804, in write_to_file
    return pyvips.Operation.call(name, self, filename,
  File "/home/john/.local/lib/python3.10/site-packages/pyvips/voperation.py", line 305, in call
    raise Error('unable to call {0}'.format(operation_name))
pyvips.error.Error: unable to call VipsForeignSaveDzFile
  VipsImage: killed for image "/home/john/pics/openslide/CMU-1.svs"
VipsImage: killed for image "/home/john/pics/openslide/CMU-1.svs"

Now that's reading a large slide image and saving as deepzoom, then canceling computation at 50%.

[yzhang525]

Thank you, John! I'll look into your example.

[yzhang525]

Hi John, I tested your code and found that it killed the process with "image = pyvips.Image.new_from_file(sys.argv[1], access="sequential")" as expected. However, when I did the way below, it didn't kill "image.write_to_file(sys.argv[2])". Any ideas? Thank you!

image = pyvips.Image.new_from_file(sys.argv[1], access="sequential")
image.write_to_file(sys.argv[2])
image.set_progress(True)
image.signal_connect('preeval', preeval_cb)
image.signal_connect('eval', eval_cb)
image.signal_connect('posteval', posteval_cb)

[jcupitt]

You need to attach the callbacks before calling write_to_file.

[yzhang525]

Sorry I wasn't clear in my last post. Here is the code I was using. My question is that it didn't kill the write_to_file.

import sys
import pyvips


def progress_print(name, progress):
    print(f'signal {name}:'.format(name))
    print(f'   run = {progress.run} (seconds of run time)')
    print(f'   eta = {progress.eta} (estimated seconds left)')
    print(f'   tpels = {progress.tpels} (total number of pels)')
    print(f'   npels = {progress.npels} (number of pels computed so far)')
    print(f'   percent = {progress.percent} (percent complete)')

def preeval_cb(image, progress):
    progress_print('preeval', progress)


def eval_cb(image, progress):
    progress_print('eval', progress)
    if progress.percent > 20:
        image.set_kill(True)


def posteval_cb(image, progress):
    progress_print('posteval', progress)

image = pyvips.Image.new_from_file(sys.argv[1], access="sequential")

image.write_to_file(sys.argv[2])
image.set_progress(True)
image.signal_connect('preeval', preeval_cb)
image.signal_connect('eval', eval_cb)
image.signal_connect('posteval', posteval_cb)

[jcupitt]

Yes, you're attaching the callbacks after you've already written the image. Try:

image.set_progress(True)
image.signal_connect('preeval', preeval_cb)
image.signal_connect('eval', eval_cb)
image.signal_connect('posteval', posteval_cb)
image.write_to_file(sys.argv[2])

[yzhang525]

By doing so, I will kill the "pyvips.Image.new_from_file", right? That's actually not what I want. I would specifically kill the process during the "writing". The reason is that in my application, the output writing (with pyramidal structure) takes extremely long time. I use "tiffsave" in my code as follows. regComplete.tiffsave(os.path.join(out_path, f"Output.ome.btf"), compression='lzw', bigtiff=True, pyramid=True, subifd=True, tile=True, depth='onetile', tile_width=512, tile_height=512)

[jcupitt]

No, it would stop the save at 20%. Try it!

[jcupitt]

new_from_file is a pixel source, so it supplies pixels down the pipeline to operations that need them. It does not loop over the image, and does no processing itself.

The write_to_file contains the pixel loop. It scans over the image dimension in sections, pulling pixels though any operations you have connected, and writing the pixels to the output. After processing each section, it sends out the eval signal. Your eval code will set kill on the pipeline and break the pixel loop in write_to_file.

[yzhang525]

I seem to understand what you mean. Let me try it out and also try with my code. Thank you, John!