"Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."

[ArchanaNeog]

Description

When concurrent TCP clients try to connect to the super socket server, a few connections fail with the error "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."

I have a concurrency test(MsTest) in which 10 TCP clients connect to the supersocket server and continue sending data for 1 minute. While running this test, I always see a few connections are closed and unable to write to the stream. However, there is no error logged in the server console. Also for every new connection superscoket logs the message "A new session connected..." as shown below. info: SuperSocketService[0] A new session connected: 63ec0218-7af5-4ea4-95e2-41adcf571004 In this scenario, the new sessions connected are less than 10 connections.

Code Sample

--Csharp SuperSockerDemo.zip contains the sample code to reproduce the issue. UnitTest1 - It has the concurrency test which creates 10 TCP clients and connects to the server. Once the connection is established it keeps sending data to the server for 1 minute Program.cs has code to host the server and SupersocketRequestProcessor.cs processes the message. The same test works as expected with other TCP server implemented without super socket. SuperSockerDemo.zip

.NET Version

.NET 8.0

Additional Information

• Operating System: Windows 10 Enterprise

[kerryjiang]

Did you use the same client test code for other TCP server implementation?

I noticed it might be the problem of your client test code.

CancellationToken should not be passed to the methods SendAsync/FlushAsync and ReceiveAsync. Because the token is triggered by timer when send/receive is happening. An task cancelled exception will be thrown in that case and the connection will drop from client side.

I also got failures in this unit test when I pass in cancellation token but it works fine after I stopped passing the cancellation token. https://github.com/kerryjiang/SuperSocket/blob/master/test/SuperSocket.Tests/PerfTest.cs

[ArchanaNeog]

Did you use the same client test code for other TCP server implementation?

I noticed it might be the problem of your client test code.

CancellationToken should not be passed to the methods SendAsync/FlushAsync and ReceiveAsync. Because the token is triggered by timer when send/receive is happening. An task cancelled exception will be thrown in that case and the connection will drop from client side.

I also got failures in this unit test when I pass in cancellation token but it works fine after I stopped passing the cancellation token. https://github.com/kerryjiang/SuperSocket/blob/master/test/SuperSocket.Tests/PerfTest.cs

The same client test code is tested for other TCP server implementation and works as expected.

The CancellationToken is triggered only when the DurationMinutes is expired as shown below.

           // Wait for the specified duration
            await Task.Delay(TimeSpan.FromMinutes(DurationMinutes), token);

            // Cancel the test
            cts.Cancel();

The test code cancels the token after 1 minute, but the exception is thrown before cts.Cancel(); is triggered. The output below shows the failure occurred before the token was canceled.

[2024-12-09T13:15:02.5121029+05:30]:Starting test... [2024-12-09T13:15:02.5478178+05:30]:An error occurred: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host... Details : at TestProject1.UnitTest1.SendMessagesAsync(CancellationToken token) in D:\WorkSpace\DicomStore-Curie\Douments\HL7ListnerForwarder\Codebase\SuperSockerDemo\TestProject1\UnitTest1.cs:line 51 [2024-12-09T13:15:02.5478206+05:30]:An error occurred: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host... Details : at TestProject1.UnitTest1.SendMessagesAsync(CancellationToken token) in D:\WorkSpace\DicomStore-Curie\Douments\HL7ListnerForwarder\Codebase\SuperSockerDemo\TestProject1\UnitTest1.cs:line 51 [2024-12-09T13:15:02.5478154+05:30]:An error occurred: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host... Details : at TestProject1.UnitTest1.SendMessagesAsync(CancellationToken token) in D:\WorkSpace\DicomStore-Curie\Douments\HL7ListnerForwarder\Codebase\SuperSockerDemo\TestProject1\UnitTest1.cs:line 51 [2024-12-09T13:16:02.5430367+05:30]:Cancelling the test... [2024-12-09T13:16:02.5436439+05:30]:Test completed.

Added a few Console.Writeline to the test code to print the test start, cancel, and complete time.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Net.Sockets;
using System.Text;

namespace TestProject1
{
    [TestClass]
    public class UnitTest1 
    {
        private const int ConcurrentConnections = 10;
        private const int DurationMinutes = 1;

        [TestMethod]
        public async Task RunTestAsync()
        {
            Console.WriteLine($"[{DateTime.Now.ToString("O")}]:Starting test...");
            CancellationTokenSource cts = new CancellationTokenSource();
            CancellationToken token = cts.Token;

            // Create concurrent connections
            Task[] tasks = new Task[ConcurrentConnections];
            for (int i = 0; i < ConcurrentConnections; i++)
            {
                tasks[i] = SendMessagesAsync(cts.Token);
            }

            // Wait for the specified duration
            await Task.Delay(TimeSpan.FromMinutes(DurationMinutes), token);

            // Cancel the test
            Console.WriteLine($"[{DateTime.Now.ToString("O")}]:Cancelling the test...");
            cts.Cancel();

            // Wait for all tasks to complete
            await Task.WhenAll(tasks);
            Console.WriteLine($"[{DateTime.Now.ToString("O")}]:Test completed.");
        }

        private async Task SendMessagesAsync(CancellationToken token)
        {
            try
            {
                using (TcpClient client = new TcpClient())
                {
                    await client.ConnectAsync("localhost", 2575); // Change the host and port as needed
                    NetworkStream stream = client.GetStream();
                    byte[] messageBytes = Encoding.ASCII.GetBytes(GenerateSampleMessage());

                    while (!token.IsCancellationRequested)
                    {
                        await stream.WriteAsync(messageBytes, 0, messageBytes.Length, token);
                        await stream.FlushAsync(token);

                        byte[] buffer = new byte[messageBytes.Length];
                        int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length, token);
                        string result = Encoding.ASCII.GetString(buffer, 0, bytesRead);
                        //Console.WriteLine($"Received message: {result}");
                    }
                }
            }
            catch (OperationCanceledException)
            {
                // Test was canceled, ignore the exception
            }
            catch (Exception ex)
            {
                Console.WriteLine($"[{DateTime.Now.ToString("O")}]:An error occurred: {ex.Message}. Details : {ex.StackTrace}");
            }
        }

        private string GenerateSampleMessage()
        {
            StringBuilder sb = new StringBuilder();
            sb.Insert(0, '\v');
            sb.Append("Sample message");
            sb.Append('\u001c');
            sb.Append('\r');
            return sb.ToString();
        }
    }
}

Also, this failure is sporadic.

[kerryjiang]

But I cannot reproduce this issue with this unit test:

https://github.com/kerryjiang/SuperSocket/blob/934acd796c918aad519cb35c64286fc5123fee2a/test/SuperSocket.Tests/PerfTest.cs#L22-L58

[ArchanaNeog]

SuperSocket/test/SuperSocket.Tests/PerfTest.cs This perfTest uses a socket instance to connect to the server. The problem is noticed when using a TcpClient to connect to the server. If I use Socket instead of TcpClient in my test code, it works as expected.

In a production environment, we won't have any control over how the client application initiates a connection to the server. The client application might use Socket or TcpClient to connect to the server as described in the below picture.

As a server, the expected behavior is, that both Socket-based or TcpClient-based clients should be able to connect to the server.

There is one more observation, the failure is noticed when using the asynchronous connect API of TcpClient ConnectAsync(string host, int port) to connect to the server. If we use synchronous connect API of the same TcpClient - Connect(string hostname, int port), it works without any issue.

[madhub]

I have also observed the same behavior when I use TcpClient connected to server written using Supersocket Note TcpClient is wrapper over socket

[kerryjiang]

Ok, let me test with TcpClient.

[kerryjiang]

The source code of TcpClient.ConnectAsync: https://github.com/dotnet/runtime/blob/3aa1ec5bb1f50f0a1bed9cfcac8734f742bcf24b/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs#L184-L189

[madhub]

The source code of TcpClient.ConnectAsync: https://github.com/dotnet/runtime/blob/3aa1ec5bb1f50f0a1bed9cfcac8734f742bcf24b/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs#L184-L189

Do you see any issue with implementation ?, I couldn't find any issues. Were you able to reproduce this with TcpClient.ConnectAsync

[kerryjiang]

@madhub No, I didn't see any problem related in that class.

@madhub I also updated the unit test to use TcpClient.ConnectAsync to connect server side, but it cannot reproduce the problem: https://github.com/kerryjiang/SuperSocket/commit/f4a12a127f0de7c77413fd46cc7937bcab886045

[kerryjiang]

@madhub Can you reproduce the problem in your local with the updated unit test?

[madhub]

@madhub Can you reproduce the problem in your local with the updated unit test?

Attached sample server & client code to reproduce the issue. Sometimes it takes several attempts to reproduce the issue. , I am running on Microsoft Windows [Version 10.0.19045.5131] , using .NET 8.0.101

1. Unzip to folder
2. dotnet build
3. run server & client

Runs the server implementation on port 2575

SuperSocketMllpServer.exe

Runs the client - make 10 concurrent connections & for 50 seconds continuously sends messages to server and read the response .

TcpTestClient.exe

SuperSocketDemoWithNfr.zip

[madhub]

@madhub Can you reproduce the problem in your local with the updated unit test?

Attached sample server & client code to reproduce the issue. Sometimes it takes several attempts to reproduce the issue. , I am running on Microsoft Windows [Version 10.0.19045.5131] , using .NET 8.0.101

1. Unzip to folder

2. dotnet build

3. run server & client

Runs the server implementation on port 2575

SuperSocketMllpServer.exe

Runs the client - make 10 concurrent connections & for 50 seconds continuously sends messages to server and read the response .

TcpTestClient.exe

SuperSocketDemoWithNfr.zip

@kerryjiang I have provided sample client & server implementation source code that reproduce the issue . Were you able to look into it ?

[madhub]

@madhub Can you reproduce the problem in your local with the updated unit test?

Attached sample server & client code to reproduce the issue. Sometimes it takes several attempts to reproduce the issue. , I am running on Microsoft Windows [Version 10.0.19045.5131] , using .NET 8.0.101

1. Unzip to folder

2. dotnet build

3. run server & client

Runs the server implementation on port 2575 SuperSocketMllpServer.exe Runs the client - make 10 concurrent connections & for 50 seconds continuously sends messages to server and read the response . TcpTestClient.exe SuperSocketDemoWithNfr.zip

@kerryjiang I have provided sample client & server implementation source code that reproduce the issue . Were you able to look into it ? @kerryjiang were you able reproduce with the client/server sample provided ?