websocketpp Crash when trying to shutdown/startup

We're having intermittent crashes. We've seen a few crashes with invalid critical sections when trying to stop our server, but recently we've had a few crash reports that seem to have happened immediately after starting our server.

We are working to try to reproduce and isolate the problem code. We will also try to see if upgrading to the latest release makes any difference. In the meantime I thought I'd post the crashing call stack in case anybody can help:

websocketpp version 0.3.0 (https://github.com/zaphoyd/websocketpp/commit/e203dbed45409111c2e95cb3e4a1d178ee57d2bc)

Boost version 1.55

Any help would be greatly appreciated, -Scott

ntdll!NtWaitForMultipleObjects+0xa
KERNELBASE!WaitForMultipleObjectsEx+0xe1
kernel32!WerpReportFaultInternal+0x581
kernel32!WerpReportFault+0x83
KERNELBASE!UnhandledExceptionFilter+0x23f
ntdll!RtlUserThreadStart$filt$0+0x3e
ntdll!_C_specific_handler+0x96
ntdll!RtlpExecuteHandlerForException+0xd
ntdll!RtlDispatchException+0x197
ntdll!KiUserExceptionDispatch+0x3a
ntdll!RtlpWaitOnCriticalSection+0xb6
ntdll!RtlpEnterCriticalSectionContended+0xa4
boost::asio::detail::win_mutex::lock+0x9 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\win_mutex.hpp @ 50]
boost::asio::detail::scoped_lock<boost::asio::detail::win_mutex>::{ctor}+0x9 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\scoped_lock.hpp @ 46]
boost::asio::detail::select_reactor::cancel_timer<boost::asio::time_traits<boost::posix_time::ptime> >+0x41 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\impl\select_reactor.hpp @ 69]
boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::cancel+0x29 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\deadline_timer_service.hpp @ 108]
boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> >::cancel+0x29 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\deadline_timer_service.hpp @ 96]
boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > >::cancel+0x6c [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\basic_deadline_timer.hpp @ 216]
websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::handle_async_shutdown+0xab [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\transport\asio\connection.hpp @ 1036]
std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>::operator()<std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> > >+0x88 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 1241]
std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>::_Do_call<boost::system::error_code const ,0,1,2,3>+0x96 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 1150]
std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,std::error_code const &),void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,std::error_code const &>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > &,std::function<void __cdecl(std::error_code const &)> &,std::_Ph<1> &>::operator()+0x13 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 1137]
std::_Callable_obj<std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,std::error_code const &),void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,std::error_code const &>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > &,std::function<void __cdecl(std::error_code const &)> &,std::_Ph<1> &>,0>::_ApplyX+0x13 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\xrefwrap @ 283]
std::_Func_impl<std::_Callable_obj<std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,std::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,std::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,0>,std::allocator<std::_Func_class<void,std::error_code const & __ptr64> >,void,std::error_code const & __ptr64>::_Do_call+0x17 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 229]
std::_Func_class<void,boost::system::error_code const &>::operator()+0x1b [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 315]
websocketpp::transport::asio::basic_socket::connection::async_shutdown+0xd7 [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\transport\asio\security\none.hpp @ 231]
websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::async_shutdown+0x2fb [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\transport\asio\connection.hpp @ 999]
websocketpp::connection<websocketpp::config::asio>::terminate+0x1aa [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\impl\connection_impl.hpp @ 1514]
websocketpp::connection<websocketpp::config::asio>::handle_close_handshake_timeout+0x151 [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\impl\connection_impl.hpp @ 1472]
std::_Func_class<void,std::error_code const &>::operator()+0x1b [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 315]
websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::handle_timer+0x13b [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\transport\asio\connection.hpp @ 341]
std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>::operator()<std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> > >+0x88 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 1241]
std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>::_Do_call<boost::system::error_code const ,0,1,2,3>+0x96 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 1150]
std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,boost::system::error_code const &),void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,boost::system::error_code const &>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > &,std::function<void __cdecl(std::error_code const &)> &,std::_Ph<1> &>::operator()+0xf [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 1137]
boost::asio::detail::binder1<std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,boost::system::error_code const &),void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const &)>,boost::system::error_code const &>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > &,std::function<void __cdecl(std::error_code const &)> &,std::_Ph<1> &>,boost::system::error_code>::operator()+0x13 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\bind_handler.hpp @ 47]
boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::system::error_code> >+0x26 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\handler_invoke_hook.hpp @ 70]
boost_asio_handler_invoke_helpers::invoke+0xb [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\handler_invoke_helpers.hpp @ 37]
boost::asio::detail::asio_handler_invoke+0xb [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\bind_handler.hpp @ 89]
boost_asio_handler_invoke_helpers::invoke+0xb [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\handler_invoke_helpers.hpp @ 37]
boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder1<std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::system::error_code> >+0x77 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\impl\strand_service.hpp @ 62]
boost::asio::io_service::strand::dispatch<boost::asio::detail::binder1<std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::system::error_code> >+0x59 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\strand.hpp @ 158]
boost::asio::detail::wrapped_handler<boost::asio::io_service::strand,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::asio::detail::is_continuation_if_running>::operator()<boost::system::error_code>+0x8a [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\wrapped_handler.hpp @ 87]
boost_asio_handler_invoke_helpers::invoke+0xf [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\handler_invoke_helpers.hpp @ 37]
boost::asio::detail::asio_handler_invoke+0xf [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\wrapped_handler.hpp @ 276]
boost_asio_handler_invoke_helpers::invoke+0xf [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\handler_invoke_helpers.hpp @ 37]
boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::asio::detail::is_continuation_if_running>,boost::system::error_code>,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_+0xd4 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\completion_handler.hpp @ 68]
boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::asio::detail::is_continuation_if_running>,boost::system::error_code>,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::+0x152 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\impl\strand_service.hpp @ 89]
boost::asio::io_service::strand::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::asio::detail::is_continuation_if_running>,boost::system::error_code>,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posi+0x84 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\strand.hpp @ 158]
boost::asio::detail::asio_handler_invoke+0x3a [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\wrapped_handler.hpp @ 234]
boost_asio_handler_invoke_helpers::invoke+0x3a [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\handler_invoke_helpers.hpp @ 37]
boost::asio::detail::wait_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64) __ptr64,void,websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > >,std::function<void __cdecl(std::error_code const & __ptr64)>,boost::system::error_code const & __ptr64>,std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >,std::shared_ptr<boost::asio::basic_deadline_timer<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime>,boost::asio::deadline_timer_service<boost::posix_time::ptime,boost::asio::time_traits<boost::posix_time::ptime> > > > & __ptr64,std::function<void __cdecl(std::error_code const & __ptr64)> & __ptr64,std::_Ph<1> & __ptr64>,boost::asio::detail::is_continuation_if_running> >::do_complete+0x116 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\wait_handler.hpp @ 70]
boost::asio::detail::task_io_service_operation::complete+0xf [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\task_io_service_operation.hpp @ 38]
boost::asio::detail::task_io_service::do_run_one+0x241 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\impl\task_io_service.ipp @ 384]
boost::asio::detail::task_io_service::run+0x127 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\detail\impl\task_io_service.ipp @ 153]
boost::asio::io_service::run+0x40 [f:\builds\99\am\8_9_release\sources\boost\source\boost\asio\impl\io_service.ipp @ 60]
websocketpp::transport::asio::endpoint<websocketpp::config::asio::transport_config>::run+0x9 [f:\builds\99\am\8_9_release\sources\websocket\source\include\websocketpp\transport\asio\endpoint.hpp @ 561]
EventsServer::run+0x129
UcaCommsServerPlugin::CommsServerThread+0xc3
kernel32!BaseThreadInitThunk+0x22
ntdll!RtlUserThreadStart+0x34

Mar 25 '15 20:03 ScottLangham

Ok. Our code was attempting to keep a list of open connection handles so that they could be closed cleanly on shutdown. But, the problem was that the connection handles were being recorded in on_message instead of on_open. (The handles were recorded along with some other info received in the first message). So, every once in a while a client would connect causing on_open to run, but before the client had sent a first message, the server would start shutting down. As on_message hadn't yet been called, this connection handle hadn't been recorded. Therefore the shutdown code missed this open connection when going round telling connections to close.

As there was still an open connection. 'run' didn't return. We timed out waiting for it to return, and ended up deleting all of the websocketpp/asio objects. Then the 'run' thread went bang as all the data it was using had been deleted.

So! If you want a clean shutdown, make sure you record connection handles in on_open.

P.S. It would be very nice if websocketpp could manage to shut itself down without external code being required to record the list of open connections and go round closing them one by one. When doing my first project with websocketpp I have spent far more time trying to work out how to get my server to stop cleanly than I have actually spent setting up the client and server to sucessfully send, receive and process messages!

Mar 25 '15 23:03 ScottLangham

The reason that WebSocket++ does not handle keeping track of outstanding connections internally is that doing so adds overhead (requires centralized locking and potentially non O(1) data structures) which affects performance for servers running 10s or 100s of thousands of connections that don't need that functionality.

If there is enough demand I might add an optional connection management policy. For the moment though I've tried to address this issue by providing example & tutorial applications that cleanly deal with the case of keeping track of and cleaning up outstanding connections.

Mar 26 '15 12:03 zaphoyd

Hi, Thanks very much for getting back to me promptly. The fix I discussed earlier has managed to make the server shutdown cleanly more often. It still does intermittently fail to shutdown, but less often now.

I think my problem is similar to this one: https://github.com/zaphoyd/websocketpp/issues/399

After doing the following steps.... (1) stop_listening (2) loop around open connections telling them to close (3) calling stop on the endpoint .... sometimes the run method doesn't return, and I can still get a new callback to my on_open handler.

The comment on line 125 of server_endpoint.cpp says: * Refer to documentation for the transport policy you are using for * instructions on how to stop this acceptance loop. But, I'm not sure where to find that documentation. It appears that new connections are still accepted after stop_listening is called. Do I also need to somehow call something else to stop accepting new connections?

Hmm, writing this has given me an idea. In on_open, I will test if the server is_listening, and if not, will tell the just opened connection to close itself. Yay! This seems to be working in my test harness.

It doesn't feel right though that after calling stop_listening, I still have to watch out for new connections being opened and tell them to go away.

Regards, -Scott

Apr 02 '15 11:04 ScottLangham

You should not get new connections after endpoint::stop_listening is called. Is there any chance your endpoint is multithreaded? If so there might be a race condition to investigate. Is the behavior you see that a handful of connections leak through after then stop? or that all connection requests continue indefinitely even after stop_listening?

You should not have to call endpoint::stop at any point. Generally you will either use stop_listening and stop connections (gracefully wind down everything) OR call endpoint::stop (forcibly disconnect everything).

Apr 02 '15 11:04 zaphoyd

My test harness program (where I observed the problem) initializes the server endpoint and starts one thread which calls run(). The main thread then sleeps for 100 milliseconds and after waking up calls stop_listening, and gracefully closes all connections. Then it waits for the run thread that was created to end (by returning gracefully from its thread function). It loops doing that forever.

Yes, I guess I shouldn't need endpoint::stop, but I added that 'for good measure' when I realised something didn't seem to be closing gracefully.

The test harness client spins around (I think my colleague adapted it from the testee_client sample), and opens connections to the server.

I observe that sometimes the run() call in the server never returns. Also, I added a bool flag called shuttingDown which the main thread sets to false at the start of its loop, it sets it to true after it calls stop_listening and gracefully closes open connections. In the on_open handler, I added: assert(!shuttingDown) as I'm not expecting any calls to on_open after calling stop_listening. But occassionally (I guess it depends what the test harness client is doing) I have seen this assert fire once.

When I changed on_open to do: if(shuttingDown) { endpoint.close(endpoint.(get_con_from_hdl(hdl), status::going_away, ""); } then I do now seem to be able to get the run() call to return.

I agree that there is probably a race!

Apr 02 '15 16:04 ScottLangham

Hi zaphoyd,

Thanks for your comments. You mentioned I shouldn't need to both close connections gracefully and call endpoint::stop. However, even after closing connections that arrive at on_open after calling stop_listening, I was still seeing infrequent hangs where the run method didn't return. To solve this: after closing connections gracefully, the code waits for the run thread to finish with a 10 second timeout. If the run thread doesn't finish, I call endpoint:stop to forcefully disconnect everything and go back to waiting for the run thread to finish for another 10 seconds. Every so often, the first timeout expires. But, after calling stop, the run thread does now seem to be finishing ok.

We are currently running some more automated tests on our entire system. I'm hopeful that with these two workarounds we'll be able to shutdown and restart our server in a controlled way with no hangs or crashes.

Regards, Scott

Apr 07 '15 09:04 ScottLangham

I've reviewed a bit more. Connections that are in the connecting state when stop_listening() is called will continue to the open state and have the open handler called after stop listening was called. As a result the following situation can happen (even in a single threaded system):

New connection ABC handshake starts
App level "stop server" called, results in endpoint::stop_listening being called and all connections in the connection list at that time being asked to close.
All connections (except ABC are closed)
Connection ABC handshake completes and the open handler is called, adding it to the connection list
Run continues as long as connection ABC is alive.

The app-level fix to this is to do as you suggest and have an additional check in the open handler for whether or not the app called its "stop server" method.

A potentially useful behavior for WebSocket++ would be for it to issue a fail rather than an open if the endpoint isn't listening once the handshake is done. This is not trivial to do right now because connections do not know which endpoint created them and endpoints do not know which connections they have created. The goal of that situation is to improve multi-threaded performance by eliminating all that centralized bookkeeping that required locks.

That said, this specific situation is pretty irritating so I'll see if I can find a way to safely but efficiently prevent outstanding handshakes from succeeding after stop_listening() is called. I've opened issue #420 to track this aspect.

With respect to "sometimes run doesn't end" even after the fix you describe... That absolutely shouldn't be happening at all. If you need to call endpoint::stop any time other than immediately before you are exiting your program then there is a bug in WebSocket++. If you find any more information about what is keeping your io_service alive please let me know.

Some things folks have had issues with in the past have been forgetting about other things they added to the io_service (like timers and signal handlers). Those sorts of things will also prevent run from exiting as it will have outstanding (non websocket++ related) work.

Apr 14 '15 13:04 zaphoyd

The FAQ said:

the connections will still be active until the io_service gets around to asyncronously processing the socket and WebSocket protocol closing handshakes.

So we cannot recycle resources about websocketpp BEFORE asio exit, it makes external io_service meanless. Not only for connection, all members like alog, elog would invalid after 'server' or 'client' destroyed.

Let's force on order of lifecycle which to avoid crash: asio::io_context -> some others based on ioctx -> websocketPP::new_server_ptr -> io_ctx stopped -> websocketPP::~server_ptr

It seems weird to move ownership of io_context to websocketPP, and make it private, but it should be shared with others components in user codes.

Proposal Add an external function std::future<void> wait_shutdown().

Sep 28 '22 16:09 vrqq