twemproxy icon indicating copy to clipboard operation
twemproxy copied to clipboard

OOM on the mbuf freelist (100+GB)

Open inter169 opened this issue 6 years ago • 4 comments

the nutcracker process consumed 100+GB phisycal memory on my production box after a data migration from another redis to this one (nutcracker). and the gdb console showed below:

(gdb) p nfree_mbufq
$1 = 6365614
(gdb) p mbuf_chunk_size
$1 = 16384

the memory consumption was nfree_mbufq * mbuf_chunk_size = 101GB approx. I have read some code fixes (pr)s about the similar phenomenon, like: https://github.com/twitter/twemproxy/pull/461 https://github.com/twitter/twemproxy/issues/203

but such fixes didn't set the limitation of the mbuf chunks, so the OOM was still here, I coded a fix, and the nutcracker can pass a command param ('-n ', in my fix) to set the max number of mbuf chunks, once exceeded the limitation it can free the mbuf after processing one req immediately.

inter169 avatar Oct 02 '17 03:10 inter169