openssl performance issues with openssl

When I was developing an https server, I encountered a problem, I created multiple threads to handle https connections , there are about 48 worker threads, each thread maintains its own SSL_CTX, and the SSL object is only access by one Thread , but when the CPS(Connection Per Second) is too large, the connection processing timeout will always occur, and the cpu only reaches about 30%. I saw the thread stack where the timeout occurred, and it basically stay at the malloc function, and the handshake has been going on for 30 seconds. Is this due to malloc's own lock, which causes openssl to not achieve good results in multi-threaded scenarios.

Thread 10 (LWP 417471):
#0  0x00007f19a97a5562 in ?? () from /usr/lib/libc.so.6
#1  0x00007f19a97a689d in ?? () from /usr/lib/libc.so.6
#2  0x00007f19a97a7b0d in malloc () from /usr/lib/libc.so.6
#3  0x00007f19aa909a90 in BUF_MEM_grow () from /usr/lib/libcrypto.so.1.1
#4  0x00007f19aad5c30c in state_machine () from /usr/lib/libssl.so.1.1
#5  0x00007f19aad470b8 in SSL_do_handshake () from /usr/lib/libssl.so.1.1
#6  0x00007f19aad48989 in SSL_read_early_data () from /usr/lib/libssl.so.1.1
#7  0x00000000004ce5cb in ssl_try_early_data (F=0x7f1971f6c840) at src/ssl.c:2296
#8  ssl_connection_establish (F=F@entry=0x7f1971f6c840) at src/ssl.c:2447
#9  0x0000000000439399 in open_ssl_connection (ssl=0x7f14aa754bb0, F=0x7f1971f6c840, type=1, fd=345575) at src/worker.c:4418
#10 fd_session_open (fd=345575, type=1, conn=<optimized out>, ssl=0x7f14aa754bb0, connecting=<optimized out>) at src/worker.c:4649
#11 0x000000000043a9f3 in init_client_connection (fd=fd@entry=345575, timer_ctx=0x0) at src/worker.c:1487
---Type <return> to continue, or q <return> to quit---
#12 0x000000000043b6cc in accept_2_work (fd=296, tm_ctx=<optimized out>, p_msec=<optimized out>) at src/worker.c:310


Thread 9 (LWP 417470):
#0  0x00007f19a987e307 in ?? () from /usr/lib/libc.so.6
#1  0x00007f19aa909ab5 in BUF_MEM_grow () from /usr/lib/libcrypto.so.1.1
#2  0x00007f19aad5c30c in state_machine () from /usr/lib/libssl.so.1.1
#3  0x00007f19aad470b8 in SSL_do_handshake () from /usr/lib/libssl.so.1.1
#4  0x00007f19aad48989 in SSL_read_early_data () from /usr/lib/libssl.so.1.1
#5  0x00000000004ce5cb in ssl_try_early_data (F=0x7f1971f6c250) at src/ssl.c:2296
#6  ssl_connection_establish (F=F@entry=0x7f1971f6c250) at src/ssl.c:2447
#7  0x0000000000439399 in open_ssl_connection (ssl=0x7f1522fe9d90, F=0x7f1971f6c250, type=1, fd=345556) at src/worker.c:4418
#8  fd_session_open (fd=345556, type=1, conn=<optimized out>, ssl=0x7f1522fe9d90, connecting=<optimized out>) at src/worker.c:4649
#9  0x000000000043a9f3 in init_client_connection (fd=fd@entry=345556, timer_ctx=0x0) at src/worker.c:1487
#10 0x000000000043b6cc in accept_2_work (fd=125, tm_ctx=<optimized out>, p_msec=<optimized out>) at src/worker.c:310

Oct 30 '22 03:10 jay900323

Whether this issue can only be solved by multi-process?

Oct 30 '22 03:10 jay900323

You've seem to answer your question yourself. If most of the processes are stuck on malloc this looks like the issue is with concurrent mallocs. Perhaps you could try to use some alternative heap implementations such as https://github.com/microsoft/mimalloc

Oct 31 '22 13:10 t8m

Otherwise, multiple processes is the only option.

Oct 31 '22 13:10 t8m

Memory allocators are often observed as a major bottleneck down the crypto layers. Any plans in switching to more common memory pools for fixed-size objects in the future ?

Nov 03 '22 14:11 wtarreau

There are no such plans currently.

Nov 03 '22 16:11 t8m

marking as inactive, will be closed at the end of the 3.4 dev cycle barring further updates

Jun 08 '24 01:06 nhorman