An error has occurred during a TLS handshake. (wss)

[teize001]

Server: Node + ws (npm) Client: Unity3D (4.6) + websocket-sharp

Usually the connection is good. But sometimes (5-10%) you receive this message. (An error has occurred during a TLS handshake. ) When receive this message, handshake is hanged for 3 minutes.

I found the code. WebSocket.cs private void setClientStream() { ... sslStream.AuthenticateAsClient(...) <== hang ... }

It Is Secured WebSocket has this chronic problem? (.net's bug??... unity3d's old framework) (https://www.google.co.jp/webhp?sourceid=chrome-instant&ion=1&espv=2&es_th=1&ie=UTF-8#q=AuthenticateAsClient+hang)

Or, the server's the problem?

[pjc0247]

I had the same issue, and still being stucked with it.

it also happens in unity 5.3.1

[pjc0247]

Hi, did you solve the problem?

I think this is a cert issue from server.

[mkalex777]

tested it with websocket server on node.js (ws). Both protocols ws and wss works good.

[devluz]

Same issue here. Pretty sure it is a bug in the old mono windows build used in Unity!

How to trigger the bug:

1. Press the play button to start unity in the Unity Editor (I am using 5.3.4)
2. Create a websocket + set the the event callbacks
3. Call ConnectAsync()
4. OnWebsocketOnOpen is called. everything worked fine
5. Press the stop button in unity and then start again
6. This time OnWebsocketOnOpen won't be called.

Turns out the thread started by ConnectAsync gets stuck in setClientStream() on AuthenticateAsClient for up to 5 minutes. I assume unity wipes the whole mono memory and this doesn't cleanup the C++ side of things used by AuthenticateAsClient (well mono doesn't do it properly I guess). Causing the method to break after stopping and restarting.

Here is my work around that works somewhat:

            sslStream = new SslStream(
                _stream,
                false,
                conf.ServerCertificateValidationCallback,
                conf.ClientCertificateSelectionCallback);



            IAsyncResult result = sslStream.BeginAuthenticateAsClient(
                host,
                conf.ClientCertificates,
                conf.EnabledSslProtocols,
                conf.CheckCertificateRevocation, (IAsyncResult res) =>
                {

                }, null);

            bool inTime = result.AsyncWaitHandle.WaitOne(timeoutMs);

            if (inTime)
            {
                sslStream.EndAuthenticateAsClient(result);
                result.AsyncWaitHandle.Close();
                returnedInTime = true;
                //worked just fine
            }
            else
            {
                sslStream.Dispose();
                result.AsyncWaitHandle.Close();
                returnedInTime = false;
                //failed. cleanup. return error
            }

Not a pretty solution but it works. Instead of directly calling AuthenticateAsClient I use the asynchronous method to do it. Then I wait a few seconds. Does it hang? If yes I kill it. This part is wrapped in another method which first tries to connect with a low timeout. Lets say 5 seconds in timeoutMs. If nothing happens after those 5 seconds I kill it. In another method I just try to reconnect if it fails once. This time with a longer timeout. Turns out after the first attempt gets stuck and fails the second one will work. Let me know if anyone has any improvements.

[pjc0247]

@devluz as you said, It was a bug in mono/Unity. Please see the link below.

https://issuetracker.unity3d.com/issues/windows-exception-thrown-by-mono-dot-security-dot-protocol-dot-tls-dot-sslstreambase-dot-endread

[devluz]

Interesting. Thanks for the link. The comment says fixed in 5.3.6 and on the top it says 5.3.7. I am on 5.3.6 and it isn't fixed yet so I am going to hope 5.3.7 is going to come out soon.

This bug in the list seem to describe the issue exactly: https://issuetracker.unity3d.com/issues/system-dot-net-dot-security-dot-sslstream-sslstream-dot-authenticateasclient-times-out-randomly

[arleyandrada]

Workaround to TLS handshake issue:

private enum SslProtocolsHack
{
    Tls = 192,
    Tls11 = 768,
    Tls12 = 3072
}

ws.OnClose += (sender, e) =>
{
    var sslProtocolHack = (System.Security.Authentication.SslProtocols)(SslProtocolsHack.Tls12 | SslProtocolsHack.Tls11 | SslProtocolsHack.Tls);
    //TlsHandshakeFailure
    if (e.Code == 1015 && ws.SslConfiguration.EnabledSslProtocols != sslProtocolHack)
    {
        ws.SslConfiguration.EnabledSslProtocols = sslProtocolHack;
        ws.Connect();
    }
};

[devluz]

@arleyandrada I just tested it. Works great! Thank you :)

[EricCogen]

This is old but it is also happening in .net core 2.0. The above workaround by @arleyandrada is working for me at the time of writing.

[Xortrox]

This does not work in unity 2017.3.0f3

[ln-12]

Workaround to TLS handshake issue:

private enum SslProtocolsHack
{
    Tls = 192,
    Tls11 = 768,
    Tls12 = 3072
}

ws.OnClose += (sender, e) =>
{
    var sslProtocolHack = (System.Security.Authentication.SslProtocols)(SslProtocolsHack.Tls12 | SslProtocolsHack.Tls11 | SslProtocolsHack.Tls);
    //TlsHandshakeFailure
    if (e.Code == 1015 && ws.SslConfiguration.EnabledSslProtocols != sslProtocolHack)
    {
        ws.SslConfiguration.EnabledSslProtocols = sslProtocolHack;
        ws.Connect();
    }
};

Had the same issue today and your workaround is still working. Is this fixed in the library itself by now? And how do I get the WebSocketServer working in wss mode?

[Darveesh]

I think the core issue is that the code is targeting .NET 3.5. I changed the target to 4.7.1 and then in the ClientSSLConfigurationClass updated the protocol to use to TLS12 (from default). The same can be done in the ServerSSLConfigurationClass as needed.

` public ClientSslConfiguration () { _enabledSslProtocols = SslProtocols.Tls12; }

public ClientSslConfiguration (string targetHost)
{
  _targetHost = targetHost;
  _enabledSslProtocols = SslProtocols.Tls12;
}

`

This allows the client is connect to new/modern wss endpoints and was actually able to run the exmaple project hitting wss://echo.websocket.org. No workarounds required if you fix the library.

Alternatively, if you are using the Nuget which is still old code, then in your application add this bit (assuming serverUri is passed to the constructor of the WebSocket class):

if (serverUri.StartsWith("wss")) { webSocket.SslConfiguration.EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12; }

[AndreasReitberger]

This issue seems still to be a thing. Good that the workarond is still working. Thanks @arleyandrada

[lfwells]

private enum SslProtocolsHack
{
    Tls = 192,
    Tls11 = 768,
    Tls12 = 3072
}

I love and hate that this worked for me, 6 years on

[L42ARO]

Workaround to TLS handshake issue:

private enum SslProtocolsHack
{
    Tls = 192,
    Tls11 = 768,
    Tls12 = 3072
}

ws.OnClose += (sender, e) =>
{
    var sslProtocolHack = (System.Security.Authentication.SslProtocols)(SslProtocolsHack.Tls12 | SslProtocolsHack.Tls11 | SslProtocolsHack.Tls);
    //TlsHandshakeFailure
    if (e.Code == 1015 && ws.SslConfiguration.EnabledSslProtocols != sslProtocolHack)
    {
        ws.SslConfiguration.EnabledSslProtocols = sslProtocolHack;
        ws.Connect();
    }
};

Had the same issue today and your workaround is still working. Is this fixed in the library itself by now? And how do I get the WebSocketServer working in wss mode?

Yep it still works November 2023, thank you so much