Cross-Thread Delayed Free Collection Issue in mimalloc

[linzj]

Summary

There is a memory leak issue in mimalloc where large blocks freed from different threads may never be properly collected, leading to memory not being returned to the system. This occurs due to a mismatch between how cross-thread delayed frees are queued and how heap collection processes them.

Root Cause

The issue stems from the interaction between mimalloc's cross-thread freeing mechanism and heap collection logic:

1. Cross-thread delayed frees are heap-specific: When a block allocated on Thread A is freed from Thread B, the block gets added to the original allocating heap's thread_delayed_free list (not the freeing thread's heap).

2. Heap collection is single-heap focused: mi_heap_collect_ex() only processes delayed frees for the specific heap being collected, not other heaps that may exist on the same thread.

3. Multiple heaps per thread: Each thread can have multiple heaps (linked via heap->next), but collection happens independently on each heap.

Problem Flow

1. Thread A allocates large block from Heap H1
2. Thread B frees the block → block queued to H1's thread_delayed_free list
3. Thread A calls mi_heap_collect_ex(H2) or H1 gets abandoned while H2 remains active
4. H1's delayed free list is never processed → memory leak

Code Locations

Where delayed frees are queued (cross-thread)

File: src/free.c:236-239

// add to the delayed free list of this heap
mi_block_t* dfree = mi_atomic_load_ptr_relaxed(mi_block_t, &heap->thread_delayed_free);
do {
  mi_block_set_nextx(heap,block,dfree, heap->keys);
} while (!mi_atomic_cas_ptr_weak_release(mi_block_t,&heap->thread_delayed_free, &dfree, block));

Where delayed frees are processed (single heap only)

File: src/heap.c:157

// free all current thread delayed blocks.
_mi_heap_delayed_free_all(heap);  // ← Only processes the specific heap being collected

Heap collection entry point

File: src/heap.c:124-180 (mi_heap_collect_ex)

Affected Scenarios

1. Large block allocations: Blocks > MI_LARGE_OBJ_SIZE_MAX (16MB on 64-bit)
2. Cross-thread freeing: Any block freed from a different thread than allocated
3. Multiple heaps per thread: Applications using mi_heap_new() to create additional heaps
4. Thread termination: When threads exit and abandon heaps containing unprocessed delayed frees
5. More obvious in thread pool scenario, where I have 32 cores and 32 threads in the thread pool.

Reproduction

This issue is most likely to manifest in applications that:

• Use large allocations (>16MB)
• Free memory from different threads than allocated
• Use multiple custom heaps per thread
• Have high thread turnover

Potential Fix

The fix should ensure that when heap collection happens (especially during MI_ABANDON), it processes delayed frees from all relevant heaps on the thread, not just the one being abandoned.

Suggested approach: Modify mi_heap_collect_ex() to iterate through all heaps in the thread's heap list (heap->tld->heaps) and process their delayed free lists when appropriate.

Environment

• mimalloc version: Latest main branch (commit: 09a27098)
• Platform: All platforms affected
• Compilation flags: Default configuration

Impact

• Memory not returned to OS despite being "freed"
• Potential memory exhaustion in long-running applications
• RSS memory usage higher than expected
• Most severe with large allocations and multi-threaded usage patterns