sta
Met StackOverflowException Exception when running websocket-sharp client inside docker container if server process is killed suddenly.
Hi websocket-sharp experts,
Thanks for providing this websocket lib so I can use it in my project. I met some issue which I could not solve myself, so I ask for help.
Scenario: I use websocket-sharp in both my server/client part. when the socket connection was established, the server will do some job and notify client timely like the progress information, finally server will send the result to client as a serialized json string. The server is base on .net framework 4.62 and the client is base on .net core 2.1.
How to trigger the issue: After the connection is established, when I kill the server process suddenly:
- The client will not be aware of this. no event in OnError and on Event in Onclose.
- So I set a timer in server to send ping to client periodically, if client not received ping in a long time(say 3 times larger than the ping interval), client will think the socket is close and close the request in client it self.
` var interval = 10000;
timer = new Timer(callback =>
{
try
{
this.Context.WebSocket.Ping();
timer.Change(interval, Timeout.Infinite);
}
catch
{
timer.Dispose();
}
},
null, interval, Timeout.Infinite);
`
Issues: The client is running fine in windows, but if I pack the .net core client inside a docker image, whenever running it in windows container using docker in windows or in linux container with Ubuntu, the container will be terminated and it will report error like: Process is terminating due to StackOverflowException.
Analysis: I have tried to increase the stack size(default is 8M) to 800M for the linux container by: ulimit -s 819200
After increase the default stack size, the container will not terminated , but if we use 'docker stats' to see the container stats, it consume 100% CPU and almost 1GB memory and the resources are not released.
Debug: I have debug the issue in windows with VS2017, the client process will not consume such much memory after kill the server process, but the websocket client seems will recursion deep in Ext.cs ReadBytesAsync function and it might cause the issue.
Questions:
- How to solve the issue?
- If server process is killed, any notification to the client socket?
Thank you.
`string wsinit = null; if (useSSL) { wsinit = $"wss://{host}:{port}/Transcribe";
}
else
{
wsinit = $"ws://{host}:{port}/Transcribe";
}
using (var ws = new WebSocket(wsinit))
{
if (useSSL)
{
ws.SslConfiguration.EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12 | System.Security.Authentication.SslProtocols.Tls11 | System.Security.Authentication.SslProtocols.Tls;
}
ws.Log.Level = LogLevel.Trace;
ws.EmitOnPing = true;
var tcs = new TaskCompletionSource<TranscribeResult>();
DateTime lastResponseTime = DateTime.Now;
DateTime lastResponseTimePing = DateTime.Now;
ws.OnMessage += (sender, e) =>
{
if (e.IsPing)
{
Trace.TraceInformation("Ping received!\n");
lastResponseTimePing = DateTime.Now;
return;
}
if (e.IsBinary)
{
return;
}
lastResponseTime = DateTime.Now;
var pack = JsonConvert.DeserializeObject<SocketPackServer>(e.Data);
switch (pack.Type)
{
case MessageType.SID:
res.SID = pack.Info;
Trace.TraceInformation("{0} {1} {2} ", DateTime.Now, pack.Type, pack.Info);
SendInfo(Event.SID, pack.Info);
break;
case MessageType.Progress:
Trace.TraceInformation("{0} {1} {2} ", DateTime.Now, pack.Type, pack.Info);
SendInfo(Event.Progress, pack.Info);
break;
case MessageType.Result:
ws.Close();
res.waveInfo = pack.info;
res.Utts = pack.UttsResult;
res.result = true;
tcs.SetResult(res);
break;
case MessageType.Error:
Trace.TraceInformation("Server Error : {0} {1}", pack.ErrInfo.ErrCode, pack.ErrInfo.ErrMessage);
ws.Close();
res.result = false;
res.ErrCode = pack.ErrInfo.ErrCode;
res.ErrorMessage = $"ServerError: {pack.ErrInfo.ErrMessage}";
tcs.SetResult(res);
break;
default:
Trace.TraceInformation("Unknow Message type returned from server! Type {0}", pack.Type);
break;
}
};
ws.OnError += (sender, e) =>
{
res.result = false;
res.ErrorMessage = $"SocketError: {e.Message}";
tcs.SetResult(res);
};
ws.OnClose += (sender, e) =>
{
Trace.TraceInformation("Client Socket closed! {0} {1}", e.Code, e.Reason);
};
Trace.TraceInformation($"Sending audio {audioPath}");
ws.Connect();
var spack = new SocketPackClient();
spack.FileName = audioPath.Replace('\\', '_');
spack.AudioInfo = audio;
spack.Locale = locale;
spack.nSpeakers = nSpeakers;
ws.Send(JsonConvert.SerializeObject(spack));
Trace.TraceInformation("Audio sent");
while (!tcs.Task.IsCompleted)
{
if ((DateTime.Now - lastResponseTime).TotalMinutes > timeoutThreshold)
{
// if no response from transcribing service for more than timeoutThreshold min, consider failure
Trace.TraceInformation("No response from transcribing service larger than {0} minutes", timeoutThreshold);
ws.Close();
res.result = false;
res.ErrCode = ServerErrorCode.Other;
res.ErrorMessage = $"TimeoutError: No response from transcribing service larger than {timeoutThreshold} minutes";
tcs.SetResult(res);
break;
}
if ((DateTime.Now - lastResponseTimePing).TotalSeconds > 30)
{
// if no ping response from transcribing service for more than 30 seconds, consider failure
Trace.TraceInformation("No Ping response from transcribing service larger than 30 seconds...");
//ws.Close();
res.result = false;
res.ErrCode = ServerErrorCode.Other;
res.ErrorMessage = $"TimeoutError: No Ping response from transcribing service larger than 30 seconds";
tcs.SetResult(res);
break;
}
System.Threading.Thread.Sleep(1000);
}
return tcs.Task.Result;
}`
11/01/2018 08:32:06|Trace|WebSocket.close|Begin closing the connection. 11/01/2018 08:32:06|Trace|WebSocket.close|Begin closing the connection. 11/01/2018 08:32:06|Trace|WebSocket.close|Begin closing the connection. 11/01/2018 08:32:06|Trace|WebSocket.close|Begin closing the connection. ^Z11/01/2018 08:32:18|Debug|WebSocket.closeHandshake|Was clean?: False sent: True received: False 11/01/2018 08:32:18|Trace|WebSocket.close|End closing the connection. 11/01/2018 08:32:18|Debug|WebSocket.closeHandshake|Was clean?: False sent: True received: False 11/01/2018 08:32:18|Trace|WebSocket.close|End closing the connection. 11/01/2018 08:32:18|Debug|WebSocket.closeHandshake|Was clean?: False sent: True received: False 11/01/2018 08:32:18|Trace|WebSocket.close|End closing the connection. 11/01/2018 08:32:18|Debug|WebSocket.closeHandshake|Was clean?: False sent: True received: False 11/01/2018 08:32:18|Trace|WebSocket.close|End closing the connection.
This is the trace information from the websocket client when I force close the process in server.
This behavior only happened in wss connection, if use ws connection, the resource will be released and no stackoverflow Exception will happened and the memory used will lower than 100M.
Also having this issue my project is here if the author would like to try it, https://github.com/PaulGWebster/Redax.dev
To cause a stack overflow, simply allow the application to run for a minute or so then attempt to take a snapshot from debug mode in vs, alternatively let the application run for a random time between 1 minute and 1 hour and it will crash by stack overflow its self.
In case I do any more development on this and anyone wants to look back it is on commit I made it a release here: https://github.com/PaulGWebster/Redax.dev/releases/tag/96b30e7
Beware this application generates significant bandwidth upwards of 20mbit
Not sure if anyone is still facing this issue or already solved it or replaced with different package. We faced the same issue after upgrading to .NET 8.0 when handling not so large data (in order of 2 to 3 MB).
Replaced the recursion logic with Task.Run() calling an async function that implements reading the stream using await stream.ReadAsync() in while loop.
Though the issue happens only in the overloaded function that handles more than 127 bytes, calling the same async function in both cases removes BeginRead/EndRead pattern and replace with async/await pattern instead which seems to help overcome this issue.
