Potential memory leak

[esiqveland]

We are testing JVips as mainly a jpeg thumbnail processor. So far we only pass a java byte[] array, as we already hold this on the heap.

This is roughly what we do:

        byte[] contents = thumb.getBlob();
        try (VipsImage image = new VipsImage(contents, contents.length)) {
          ...
          image.colorspace(Srgb)
          image.thumbnail(...)
          image.autorot();
 
          var output = image.writeJPEGToArray(JPEG_COMPRESSION_QUALITY, true);
          return output;
       }

But when we enable the JVips processor, several instances are almost immediately going OOM:

Processing with the above code was enabled around 09:17 in the above.

Is this something you have seen with VipsImage or are we using it wrong?

Is there a way I can help track this down?

[hauf-rakuten]

Unfortunately we have the very same situation. I could improve the situation a little bit by turning of caching and multi thread with: VipsContext.setMaxCache(0); VipsContext.setConcurrency(1); but still the memory usage continuously increases. Its not the java heap that increases but System Memory. If you shutdown Libvips context VipsContext.shutdown(); System Memory is freed up but then the context is not usable anymore.

[esiqveland]

@hauf-rakuten have you tried holding one VipsContext per thread, like a threadlocal? Not sure if that is possible, but worth trying. I kind of gave up on this and just delegate to vips/vipsthumbnail as subprocesses

[hauf-rakuten]

@esiqveland thanks for your reply. Thats the point there is no way to control the VipsContext. It is initiated on the first call of any JVips class and the only thing I found you can do is to set some flags to it and close it. But most likely I have over looked something. Thats why I am asking.

[frederico-apolonia]

@hauf-rakuten were you able to find out what was causing the memory leak?

[hauf-rakuten]

At the end I was able to fix it by switching the glib memory allocator that is default for debian to mimalloc and this reduced the memory usage dramatically and also solved the issue with the memory leak. There is a known issue with multithreaded apps in glib but I am not sure if thats really the cause for the issue I saw.

[frederico-apolonia]

Thanks for your quick response, @hauf-rakuten . Did you just swap glib with mimalloc? Mimalloc doesn't have the g_object_free and other g_ functions that glib has, afaik

[hauf-rakuten]

Yes, we simply replaced glib with mimalloc. We intensively tested and also measured emory usage even with other implementations like tcmalloc and jemalloc. Haven't seen any issue like missing functions but mimalloc turned out to perform best for us.