how to release the memory

[NextGuido]

@jcupitt Hi, I'm back again! I have such code below and I want to release the memory after saving the tiff file instead of closing the program. What should I do?

import os
from tqdm import trange
from time import time, sleep
import pyvips

tiles_across = 20
tiles_down = 20
tile_width = 2448
tile_height = 2048
overlap = 229

# src path have 400 images(each image's shape i 2048*2448*3) and i wanna insert them into a big pyramid tiff
src = r'/media/ubuntu/code/code/acid-fast-staining/1000x/inotify_predict/data/Picture/FFL_100x_18-2064-1'
dst = r'pic/'
before_write = time()
img = pyvips.Image.black(1, 1, bands=3)
for y in trange(tiles_down):
    row = pyvips.Image.black(1, 1, bands=3)
    for x in range(tiles_across):
        # eg: 01-01.jpg
        filename = os.path.join(src, "{:02d}-{:02d}.jpg".format(y + 1, x + 1))
        tile = pyvips.Image.new_from_file(filename, access=pyvips.Access.SEQUENTIAL)
        row = row.insert(tile, x * (tile_width - overlap), 0, expand=True)

    img = img.insert(row, 0, y * (tile_height - overlap), expand=True)
    del row

print(img)
img.write_to_file(os.path.join(dst, 'insert_output.tif'), tile=True, pyramid=True, compression='jpeg', bigtiff=True)
after_write = time()
print('time cost: {}s'.format(after_write - before_write))

del img
while 1:
    sleep(100)

I use del method in python, but it seems to be useless...

[jcupitt]

Hey @NextGuido,

Your program seems OK, it will be freeing memory. Python probably won't return the memory back to the OS though -- it'll keep it for reuse.

Are your images always in a rectangular grid? You'll find arrayjoin uses less memory than a lot of insert calls.

https://libvips.github.io/pyvips/vimage.html#pyvips.Image.arrayjoin

You can write access="sequential", it's slightly easier.

[NextGuido]

My system is ubuntu 16.04. we can see that even if the tiff file is saved, but the memory occupation is still there. the memory could be released only if i shut down the program. I don't know the reason.

(AI) ubuntu@ubuntu-desktop:/media/ubuntu/code/code/acid-fast-staining/1000x/inotify_predict/temp-inference$ vips -v
vips-8.8.0-Fri Mar 22 15:02:53 UTC 2019

[jcupitt]

That's not a leak though, it's just not returned to the OS.

Try adding a loop. For example:

import sys
import os
from time import time
import pyvips

tiles_across = 2
tiles_down = 2
tile_width = 1450
tile_height = 2048
overlap = 229

# src has 4 1450x2048 jpg images
src = sys.argv[1]
dst = sys.argv[2]

for loops in range(10000):
    before_write = time()
    img = pyvips.Image.black(1, 1, bands=3)
    for y in range(tiles_down):
        row = pyvips.Image.black(1, 1, bands=3)
        for x in range(tiles_across):
            filename = f"{src}/{y + 1:02d}-{x + 1:02d}.jpg"
            tile = pyvips.Image.new_from_file(filename, access="sequential")
            row = row.insert(tile, x * (tile_width - overlap), 0, expand=True)
    
        img = img.insert(row, 0, y * (tile_height - overlap), expand=True)

    img.write_to_file(dst)
    after_write = time()
    print(f"{loops}: {after_write - before_write}s")

Now I can run with:

john@kiwi:~/try$ python3 mem.py sample x.tif[tile,pyramid,compression=jpeg]
0: 0.26742029190063477s
1: 0.26651477813720703s
2: 0.2679414749145508s
3: 0.2623453140258789s
4: 0.29327392578125s
5: 0.2754788398742676s
... etc.

If I watch RES with top, I see:

iteration	RES (mb)
50	98
100	100
150	102
250	101
300	99
400	103
500	103
600	102
1000	103
2000	105

So I think there's no leak and everything is working.

[NextGuido]

https://github.com/libvips/pyvips/issues/105#issuecomment-507101612 About this code(I put it into a monitoring program) I methioned before! I add an operation to new a python thread to save the pyramid tiff. Everything works well while the images count is less than 1000. But it will throw an error: "Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)" when images count is more than 1000. I found that it's ok with the max memory occupation(45G/125G) while the program went. So, may you help me with this? Many thanks here

oh, my ubuntu system is 16.04, vips version is 8.8. Next is the detailed configuration:

[jcupitt]

Could you open a new issue with a complete sample program please?