twemproxy
twemproxy copied to clipboard
OOM on the mbuf freelist (100+GB)
the nutcracker process consumed 100+GB phisycal memory on my production box after a data migration from another redis to this one (nutcracker). and the gdb console showed below:
(gdb) p nfree_mbufq
$1 = 6365614
(gdb) p mbuf_chunk_size
$1 = 16384
the memory consumption was nfree_mbufq * mbuf_chunk_size = 101GB approx. I have read some code fixes (pr)s about the similar phenomenon, like: https://github.com/twitter/twemproxy/pull/461 https://github.com/twitter/twemproxy/issues/203
but such fixes didn't set the limitation of the mbuf chunks, so the OOM was still here, I coded a fix, and the nutcracker can pass a command param ('-n ', in my fix) to set the max number of mbuf chunks, once exceeded the limitation it can free the mbuf after processing one req immediately.