Magick.NET
Magick.NET copied to clipboard
dlemstra
Memory leak on Linux - Multi-threaded application
Magick.NET version
Magick.NET-Q8-x64 -> 14.6.0
Environment (Operating system, version and so on)
Linux - Ubuntu 24.04 in WSL or Linux in docker container (base image: mcr.microsoft.com/dotnet/sdk:9.0)
Description
Hi.
I think, there is memory leak when using Magick.NET on Linux during processing images. Allocated memory is never released and increases in time. Speed of increase depends on file size.
It seems to me similar to this issue https://github.com/dlemstra/Magick.NET/issues/1163 where you somehow solved webp?
My use case: I need to process single image or each frame of multi-framed image (like GIF for example), resize it using InterpolativeResize() and this resized Image I need to use for further processing. Also I have limited resources, so I need to setup limits.
To simplify this use case I created demo, where 5 threads are processing GIF or PNG. I've also implemented a mechanism where each iteration of image processing uses a different thread, as it appears each thread maintains its own reserved memory pool. The memory leak seems to occur because when a previously idle thread begins processing images, it allocates new resources that are never properly released.
Additional information On windows, the memory management looks as expected, problem is just on Linux, where memory is never released. I was trying to play with GLIBC_TUNABLES as it was mentioned in this comment but without any success.
Expected Behavior: Memory usage should stabilize after initial allocation, similar to Windows behavior.
Actual Behavior: Memory usage continuously increases on Linux, leading to eventual out-of-memory conditions.
Thank you for your help.
Steps to Reproduce
Demo project:
-
In Resources folder, there should be PNG1.png, PNG2.png, ..., PNG5.png and GIF1.gif, ... GIF5.gif. I could not add them into zip with project so please add them manually into Resources folder.
-
There is Dockerfile -> build image using:
docker build -t image-magick-memory-leak . -
Run image with PNG or GIF CMD :
docker run -v /path/to/volume/:/tmp image-magick-memory-leak PNGordocker run -v /path/to/volume/:/tmp image-magick-memory-leak GIF -
At start you have some time to go into container and start dotmemory profiling if you want to ->
docker exec -it containerId /bin/bash->cd /tmp->/dotMemoryclt/tools/dotmemory attach 1 -
or you can just look at attached dmw files or pictures from the process done by myself
Windows - expected behavior:
windows_dmws.zip PNG:
GIF:
Linux - increasing in time:
linux_dmws.zip PNG:
GIF:
Images
Is there any way to force a library on Linux to release memory as aggressively as it does on Windows? Even at the cost of worse performance. In our case, memory is a much bigger problem than processing speed.
This looks like an issue in the internals of dotnet. It looks like it's not returning memory to the OS and due to fragmentation you are probably running into this issue. I don't know how you can control this behavior.
Had the same issue on my project recently which resizes images in a multi-threaded context.
I spent quite some time on this, trying to hunt what I was missing but in the end it's indeed just unmanaged memory growing and growing, even when forcing GC collection.
https://github.com/BenHUET/sky-tex-opti/blob/master/Services/Resizer/ImageMagickResizerService.cs
Hello. @dlemstra
Can we rule out with certainty that this is a bug in the ImageMagick.NET wrapper or in ImageMagick itself? Ideally, I would like to have bulletproof evidence that this is a bug in the dotnet runtime. Do you have any recommendations on what to test and what output could serve as evidence?
Maybe just bypass .net wrapper and do the same thing just by using C directly? Monitor with some kind of C memory profiler? Have you already done something similar based on this issue to be sure it's not a bug in ImageMagick?
I can do it myself, but it will be very difficult for me, so I just want to make sure I don't do it unnecessarily.
Thank you
You cannot rule that out but it is very unlikely because the same C# and C code is being used on both platforms. I don't know enough about the internals of dotnet to know how you can validate that the library is leaking memory. And if you want to do memory analysis you will probably need to build your own version of this library that uses memory analysis tooling. In the past I have done this directly on Windows with the ImageMagick library and there I skipped the wrapper. But here I don't see a good reason for me to spend time on this because this issue is not happening on both platforms.
Hello. @dlemstra
I think I managed to simulate the behavior when bypassing the .NET wrapper. It's pointing to an issue in ImageMagick on Linux paltform, because dotnet runtime does not play any role in this case.
I have cpp app doing the same thing as c# example provided. On Windows the process starting at 9MB, ends up with 18MB used. On Linux starting at 9MB ends up with 800MB. Logs provided.
Use case: Process of demo is the same as in original c# use case. I have 12 cores available and in 3 iterations I'm processing 1192 frames GIF in 5 threads parallel. Each iteration takes different 5 threads -> 1st iteration 1,2,3,4,5 | 2nd iteration 6,7,8,9,10 | 3rd iteration 11,12,1,2,3
Actual results: Looks like each thread has locked some amount of memory which never release.
Workaround: Probably no workaround, just reducing final memory impact -> Optimalization of loading image for example
DEV Notes: I'm using Jetbrains CLion with 2 CMake profiles -> WSL with WSL Toolchain and WIN with MinGW Toolchain
How to reproduce:
- Download demo image_magick_memory_leak_cpp.zip
- I could not include all GIF files into one ZIP so please copy GIF1.gif and create also GIF2.gif GIF3.gif GIF4.gif GIF5.gif
- Build cmake
- Run on Windows / Linux with arguments
MagickLeakTest GIF1.gif GIF2.gif GIF3.gif GIF4.gif GIF5.gif
Probably this should be related to ImageMagick project, but it's definitely an issue in .NET wrapper also from my perspective.
Please understand that I am not a C developer and have never actively worked with C, so all source codes are generated by AI. However, it seems very sensible and usable. At the same time, the memory management seems to be okay from my point of view. This is actually proven by the correct behavior on Windows
Your code uses the Wand api and this library is not using that. But at a first glance I don't see what could be causing what you are seeing. On Linux we are using a different memory manager and according to GitHub Copilot the memory that is freed is not directly being returned to the operating system:
Yes, it's absolutely possible and quite common for memory managers on Linux to not immediately return memory to the OS when free() is called. This is actually the default behavior in most cases. Here are the key reasons why: Memory Manager Behavior
Glibc malloc (ptmalloc2) - the default allocator on most Linux systems:
- Maintains freed memory in internal pools/freelists
- Only returns memory to the OS under specific conditions:
- When a large contiguous block at the top of the heap can be trimmed
- When using malloc_trim() explicitly
- When the heap grows beyond certain thresholds and then shrinks significantly
Why this happens:
- Performance optimization - Allocating/deallocating from the OS is expensive
- Fragmentation - Small freed blocks scattered throughout the heap cannot be easily returned
- Future allocations - Keeping memory available for quick reuse
When Memory IS Returned to OS
- Large allocations (typically >128KB) that use mmap() are returned immediately when freed
- Heap trimming occurs automatically in some cases or when malloc_trim(0) is called
- Process termination - all memory is returned to the OS
--- snip ---
Alternative Allocators
If immediate memory return is critical, consider:
- jemalloc - often returns memory more aggressively
- tcmalloc - Google's allocator with different policies
- Custom allocators designed for specific use cases
This behavior is by design and generally beneficial for performance, but it can be surprising when monitoring memory usage!
Can you confirm that either calling malloc_trim or using a different allocator solves your issue. At first though it feels like I should add a method that will allow you to call malloc_trim and that I should not use a different allocator because of the points mentioned above.
Actually, I didn't try any different allocator, you mentioned, as calling malloc_trim(0) looks very promising. Linux process ends up with 10MB memory used.
Final rows of the log file, where malloc_trim(0) do exactly what we want.
...
Thread 2 (logical 12) | Iteration 3/3: Memory trimmed. - Memory Usage: 486 MB
Thread 1 (logical 11) | Iteration 3/3: frame 1192/1192 Processing finished. Frame destroyed - Memory Usage: 486 MB
Thread 3 (logical 1) | Iteration 3/3: frame 1190/1192 Processing finished. Frame destroyed - Memory Usage: 482 MB
Thread 2 (logical 12) | Iteration 3/3: frame 1192/1192 Processing finished. Frame destroyed - Memory Usage: 468 MB
Thread 1 (logical 11) | After processing complete - Memory Usage: 462 MB
INFO: Calling malloc_trim(0)...
INFO: malloc_trim returned: 1
Thread 3 (logical 1) | Iteration 3/3: frame 1191/1192 Processing finished. Frame destroyed - Memory Usage: 312 MB
INFO: Calling malloc_trim(0)...
INFO: malloc_trim returned: 1
Thread 3 (logical 1) | Iteration 3/3: Memory trimmed. - Memory Usage: 309 MB
Thread 2 (logical 12) | After processing complete - Memory Usage: 303 MB
INFO: Calling malloc_trim(0)...
INFO: malloc_trim returned: 1
Thread 3 (logical 1) | Iteration 3/3: frame 1192/1192 Processing finished. Frame destroyed - Memory Usage: 169 MB
Thread 3 (logical 1) | After processing complete - Memory Usage: 144 MB
INFO: Calling malloc_trim(0)...
INFO: malloc_trim returned: 1
INFO: Forcing garbage collection...
INFO: Calling malloc_trim(0)...
INFO: malloc_trim returned: 1
INFO: Garbage collection completed
Finished iteration: 3/3.
-------------------------------------------------------------------
Thread 0 (logical 0) | Processing finished - Memory Usage: 11 MB
=== FINAL CLEANUP ===
INFO: Forcing garbage collection...
INFO: Calling malloc_trim(0)...
INFO: malloc_trim returned: 1
INFO: Garbage collection completed
Thread 0 (logical 0) | Final cleanup complete - Memory Usage: 10 MB
Process finished with exit code 0
Updated code:
There is following class with static trim_memory and force_gc functions. trim_memory is called after each 10 iterations and after process + trim_memory and force_gc at the end during final cleanup.
// Memory management utilities
class MemoryManager {
public:
static void trim_memory() {
#ifdef __linux__
// Linux - malloc_trim
printf("INFO: Calling malloc_trim(0)...\n");
int result = malloc_trim(0);
printf("INFO: malloc_trim returned: %d\n", result);
#elif defined(_WIN32)
// Windows - HeapCompact
printf("INFO: Calling HeapCompact for all heaps...\n");
HANDLE heaps[64];
DWORD num_heaps = GetProcessHeaps(64, heaps);
for (DWORD i = 0; i < num_heaps; i++) {
SIZE_T freed = HeapCompact(heaps[i], 0);
printf("INFO: HeapCompact on heap %lu freed: %zu bytes\n", i, freed);
}
#else
printf("INFO: Memory trimming not supported on this platform\n");
#endif
}
static void force_gc() {
printf("INFO: Forcing garbage collection...\n");
trim_memory();
// Other memory optimization
#ifdef __linux__
// Sync file system caches
sync();
// Pokusit se uvolnit page cache (vyžaduje root)
// system("echo 1 > /proc/sys/vm/drop_caches");
#endif
printf("INFO: Garbage collection completed\n");
}
};
Hello,
@dlemstra
What do you think about the results? What will be the next step and planned solution from the .NET wrapper perspective?
If you'll need some assistance form my side, just let me know. At least, if you'll have some POC I can test it easily as I have the demo prepared for this exact use case.
@lkratochvil I tried your solution in my ASP.NET Core app. I made an endpoint that calls malloc_trim(0). When memory keeps growing and doesn't drop after image processing is done, calling this fixes it - memory goes back to normal levels. So yes, it works.
But I'm not sure - is this really the right way to fix the problem?
I don't know if there will be any next steps. This apparently is something inside the default memory allocator on Linux. Maybe there are other ways than calling malloc_trim(0) like setting environment variables to make sure trimming happens.
I don't know if there will be any next steps. This apparently is something inside the default memory allocator on Linux. Maybe there are other ways than calling
malloc_trim(0)like setting environment variables to make sure trimming happens.
I set ENV GLIBC_TUNABLES=glibc.malloc.trim_threshold=65000 in my Docker container, and it helps a bit. The memory usage doesn't go too high, but when the system is idle, it stays around 20-25% instead of dropping to the expected 6-10% (which is normal when the app just checks image details from the database).To improve this, I added a background service that runs malloc_trim(0) every 5 minutes. With this change, memory usage drops to around 15%, which is better but still not perfect.
After making all these changes, I also checked the memory statistics.
{
"processMemory": {
"workingSet64": 162414592,
"privateMemorySize64": 281866240,
"virtualMemorySize64": 5392048128,
"pagedMemorySize64": 0,
"pagedSystemMemorySize64": 0,
"nonpagedSystemMemorySize64": 0,
"peakWorkingSet64": 519966720,
"peakVirtualMemorySize64": 5763731456,
"peakPagedMemorySize64": 0
},
"gcMemory": {
"totalAllocatedBytes": 532916176,
"totalMemory": 15863824,
"collectionCounts": {
"gen0": 35,
"gen1": 35,
"gen2": 34
},
"memoryInfo": {
"highMemoryLoadThresholdBytes": 483183820,
"memoryLoadBytes": 155692564,
"totalAvailableMemoryBytes": 402653184,
"heapSizeBytes": 15718424,
"fragmentedBytes": 513952,
"index": 35,
"generation": 2,
"compaction": false,
"concurrent": false,
"pinnedObjectsCount": 2,
"finalizationPendingCount": 4
},
"isServerGC": true,
"largeObjectHeapCompactionMode": "Default"
}
}
Analysis:
- Large Private Memory (269MB) with relatively small GC heap (15.8MB) suggests:
- Image data is likely in unmanaged memory (GDI+, WIC, or DirectX)
- Possible bitmap handles not being disposed
- High Gen2 collections may indicate:
- Image wrappers surviving collections
- Caching mechanisms holding references
The nearly 250MB gap between private memory and GC heap is likely undisposed image resources.
FWIW I experienced this as well on a dotnet8 SDK linux image. Seemed that even disposing the ImageMagick image or related streams did not stop it from accumulating memory. I had been using docker stats and seeing the memory usage increase as I was processing tens of images (50), and the app had gained ~200 MB and was growing per call.
As @yauhenibhIDT mentioned you can reduce the allocated memory with malloc_trim(0). The next release will add TrimMemory to the ResourceLimits class that will execute malloc_trim on supported platforms. I will close this issue because I haven't seen any proof yet that this library contains a memory leak. It might still be possible that there is a memory leak and if someone finds one then please open up a new issue with a reproducible example that demonstrates this issue.