IXWebSocketServer hangs on OpenSSL handshake

[rossum2]

WebSocket server occasionally stops accepting new connections probably when SSL handshake is interrupted in the right moment. I use OpenSSL, so IXSocketOpenSSL.cpp is relevant here. It is possible that this is not IXWebSocket bug but an OpenSSL bug, but if it is so, is there an OpenSSL version one should avoid? This is a rare thing to happen, so I wasn't lucky enough to catch it on both client and server side. Here is a backtrace from gdb:

 (gdb) bt
#0  0x00007f2b99edf8e4 in read () from /lib64/libpthread.so.0
#1  0x00007f2b9a1a1132 in sock_read () from /lib64/libcrypto.so.1.1
#2  0x00007f2b9a19a85e in bread_conv () from /lib64/libcrypto.so.1.1
#3  0x00007f2b9a1996c3 in bio_read_intern () from /lib64/libcrypto.so.1.1
#4  0x00007f2b9a199c77 in BIO_read () from /lib64/libcrypto.so.1.1
#5  0x00007f2b99a4836f in ssl3_read_n () from /lib64/libssl.so.1.1
#6  0x00007f2b99a4c63e in ssl3_get_record () from /lib64/libssl.so.1.1
#7  0x00007f2b99a49f40 in ssl3_read_bytes () from /lib64/libssl.so.1.1
#8  0x00007f2b99a7e19d in tls_get_message_header () from /lib64/libssl.so.1.1
#9  0x00007f2b99a73938 in state_machine.part () from /lib64/libssl.so.1.1
#10 0x00007f2b99a5ec98 in SSL_do_handshake () from /lib64/libssl.so.1.1
#11 0x00000000005d30ad in ix::SocketOpenSSL::openSSLServerHandshake(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) ()
#12 0x00000000005d420d in ix::SocketOpenSSL::accept(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) ()
#13 0x00000000005a87c3 in ix::SocketServer::run() ()
#14 0x00000000005aa384 in void std::__invoke_impl<void, void (ix::SocketServer::*)(), ix::SocketServer*>(std::__invoke_memfun_deref, void (ix::SocketServer::*&&)(), ix::SocketServer*&&) ()
#15 0x00000000005a9712 in std::__invoke_result<void (ix::SocketServer::*)(), ix::SocketServer*>::type std::__invoke<void (ix::SocketServer::*)(), ix::SocketServer*>(void (ix::SocketServer::*&&)(), ix::SocketServer*&&) ()
#16 0x00000000005abd67 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)())) std::thread::_Invoker<std::tuple<void (ix::SocketServer::*)(), ix::SocketServer*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) ()
#17 0x00000000005abc9a in std::thread::_Invoker<std::tuple<void (ix::SocketServer::*)(), ix::SocketServer*> >::operator()() ()
#18 0x00000000005abc64 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (ix::SocketServer::*)(), ix::SocketServer*> > >::_M_run() ()
#19 0x00007f2b99750ba3 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#20 0x00007f2b99ed614a in start_thread () from /lib64/libpthread.so.0
#21 0x00007f2b98e2df23 in clone () from /lib64/libc.so.6
(gdb) info threads
  Id   Target Id                                           Frame 
  1    Thread 0x7f2b9b913500 (LWP 93887) "main_thread"    0x00007f2b99edc2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7f2b98712700 (LWP 93888) "periodic_operations"    0x00007f2b99edffb0 in nanosleep () from /lib64/libpthread.so.0
* 3    Thread 0x7f2b97f11700 (LWP 93889) "SocketServer::a" 0x00007f2b99edf8e4 in read () from /lib64/libpthread.so.0
  4    Thread 0x7f2b97710700 (LWP 93890) "SocketServer::G" 0x00007f2b99edc2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5    Thread 0x7f2b96f0f700 (LWP 182437) ""               0x00007f2b98e22ca1 in poll () from /lib64/libc.so.6

[bsergean]

It's hard to tell what's going on, I had trouble a while ago with TLS version 3, and upgrading to a more recent version of openssl fixed it.

You could try to build the library with thread sanitizer or address sanitizer on, and see if you can repro the problem that way, in case there is a data race or similar error.

[bsergean]

Last thought is that you could try to build/link with mbedtls too and see if you can reproduce the hang there.