How to limit pyvips RAM usage

[mirko0x5f]

Hello, I'm developing a program that will be run on k8s as a pod. I have set a limit for the pod RAM capabilities, so that k8s will kill that pod if it exceeds said limits; I would like to know if there is a way to limit the pyvips RAM utilization so that I can gracefully quit / throw an exception instead of having the container being killed by k8s.

I tried utilizing resource.setrlimit from the resource python package but the program seems to just crash during malloc or throw seg. faults.

I'm looking for some way of letting pyvips "know" that it has only that much memory available or to know how much memory will a given pyvips operation consume, before doing it.

Thank you for your time, Mirko

[jcupitt]

Hello, you can limit the pyvips operation cache by memory usage.

pyvips.cache_set_max_mem(1024 * 1024)

Sets it to 1MB. The default is 100MB.

This only affects the cache. If a pipeline must use a lot of memory to execute, it'll still use a lot of memory.

More threads use more memory. By default, pyvips will make a thread for every hardware thread. If your processor has hyperthreading, this can use a lot of memory without giving a useful speedup. You could consider lowering the number of threads. I would do some benchmarking to find a good value for your workload.

You can also optimise your pipeline for lower memory use. If you post some sample code I could have a look.

You can enable pyvips memory tracking with:

pyvips.leak_set(True)

It'll print the memory high-water mark on exit. This can be useful for testing.

[jcupitt]

For example, if I add leak_set to the watermark example:

john@kiwi:~/GIT/pyvips/examples (master)$ VIPS_CONCURRENCY=4 ./watermark.py ~/pics/k2.jpg x.jpg hello
memory: high-water mark 7.01 MB
john@kiwi:~/GIT/pyvips/examples (master)$ VIPS_CONCURRENCY=1 ./watermark.py ~/pics/k2.jpg x.jpg hello
memory: high-water mark 3.76 MB

Note that the reported memory usage is only for libvips pixel buffers.