Consistent state across forks

[davidchisnall]

I am increasingly unconvinced that snmalloc is safe to use in programs that call fork. I had originally thought that the mostly lockless design meant that we're safe, but I think there are three problems:

• If a thread holds any of the range locks, we can deadlock on fork.
• Any allocators owned by threads other than the one calling fork will leak.
• Any allocators owned by threads other than the one calling fork will be in an inconsistent state.

This is all bad.

The first problem can be solved by acquiring the lock(s?) in the prepare callback for pthread_atfork and then releasing it in both the parent and child callbacks.

The second can be solved by putting all allocated allocators in a linked list and walking that list in the child to return them all to the pool (and probably coalescing them at that point).

I'm not sure about the third. Do we atomically pop things off free lists on the fast path? On anything that's doing read-modify-write operations on an allocator, we'd need an asymmetric barrier.

I'm not especially worried about this because any program that calls malloc from a multithreaded program that calls fork and not execve is going to leak memory, but ideally malloc between fork and execve should work (which requires fixing the first one).

[mjp41]

Three other places I can think of in addition

• the allocator pool has a lock as well. So needs fixing in the same way.
• the singleton pattern uses a lock too.
• message queue is non-linearisable. This means you can begin an enqueue, another thread forks, and then the message queue will be in a state where it will never become reconnected to the back. The child post fork handler should process the message queue and reconnect the back if required.