BedrockFramework icon indicating copy to clipboard operation
BedrockFramework copied to clipboard

Exception when to disconnect.

Open juliolitwin opened this issue 4 years ago • 14 comments

Hi,

I do not know if it is a case that was already expected, but curious about Bedrock, I developed a simple client that can connect 500 simultaneous clients and send packets, so far everything is happening normally, but when I close the client (forcing the server to lose connection with all clients), it simply generates exceptions for each disconnection, but the problem is that it ends up making the server slow with so many exceptions, without being able to do new actions, such as accepting new connections or new packets.

I'm forgetting something? Cheers.

juliolitwin avatar Apr 23 '20 23:04 juliolitwin

How did you close the connection? You get an exception if there's a TCP reset but not a FIN.

davidfowl avatar Apr 28 '20 03:04 davidfowl

I just force my client to close. It's gave me reset and "An existing connection was forcibly closed by the remote host."

Force closing is not the correct way, but in a normal socket these problems do not happen, is there any way to prevent me from the BedRock being stopped while disconnecting the connections and with the high use of cpu (because it is generating many exceptions)?

Because if there is no solution, anyone can attack by creating a simple client and try to connect several clients and then force the closing and cause this problem on the server.

With IntelliTrace using the 'snapshots (managed and native)' option, it is possible to see exceptions more clearly.

juliolitwin avatar Apr 28 '20 13:04 juliolitwin

@juliolitwin Do you see high CPU repeatedly or do you see it once? Did you run a single performance test where clients connect and all disconnect or did you do it multiple times?

davidfowl avatar Apr 28 '20 14:04 davidfowl

@davidfowl Whenever I force clients to close, exceptions occur and the CPU increases.

I recorded a GIF to show you, I'm using 1000 connections to be more visible and after I close the client, I wait a moment for you to see that the CPU process is "stopped" at a point in the diagnosis and it goes up to the high point and is stopped even when all clients are disconnected and gives all exceptions. Then I open the client again, just to show that the server is unviable, it cannot accept new connections because it is still processing.

Gif: https://gifyu.com/image/l5vv

juliolitwin avatar Apr 28 '20 15:04 juliolitwin

I've seen similar behavior before, but mostly on the first set of disconnects. Try this:

  • Run in release mode
  • Run without logging at all (you have debug logging on in that sample).
  • Do multiple runs with the same server process running and check the CPU.

davidfowl avatar Apr 28 '20 15:04 davidfowl

Oops, in the gif I enabled debugging just for you to see the exceptions, but with the log disabled and in the release mode it still keep giving samething.

About running the client several times, the cpu has no effect, as it does not accept any connections, it is as if the network thread / async was totally busy processing the exceptions and my client only return me saying can't to connect.

juliolitwin avatar Apr 28 '20 15:04 juliolitwin

About running the client several times, the cpu has no effect, as it does not accept any connections, it is as if the network thread / async was totally busy processing the exceptions and my client only return me saying can't to connect.

Wait until the clients disconnect, then try again and do another run with the server still running and a new set of clients. What I'm asking is to run the test with clients multiple times with the same server instance.

davidfowl avatar Apr 28 '20 15:04 davidfowl

I decrease the number of connections because it takes too long to disconnect all clients and also decreases the rate of packets sent. I'm using server in release and record two versions with log and without log.

GIF (no log): https://gifyu.com/image/l5nW

GIF (with log): https://gifyu.com/image/l5Wm

Strange that with 1000 connections, when I close and wait for the server to process and then try to open the client is unable to make a connection.

juliolitwin avatar Apr 28 '20 16:04 juliolitwin

Can you also stop debugging? That slows things down significantly.

davidfowl avatar Apr 30 '20 17:04 davidfowl

It really made a drastic difference.

juliolitwin avatar May 01 '20 17:05 juliolitwin

@davidfowl Even though BedrockFramework using aspnetcore as a base, do you already consider the project ready for production use? Are there any recommendation to limit connections per socket?

juliolitwin avatar May 10 '20 14:05 juliolitwin

@davidfowl Even though BedrockFramework using aspnetcore as a base, do you already consider the project ready for production use?

Yes it's pretty solid. There are still some things I'd like to bring over from experimental before shipping but haven't had time to spend on it as yet.

Are there any recommendation to limit connections per socket?

I assume you mean limit general connections? There's nothing in here but it's trivial to add.

davidfowl avatar May 22 '20 05:05 davidfowl

In MyCustomProtocol in readasync, I can't get the received packet array, as in the example on the line:

https://github.com/davidfowl/BedrockFramework/blob/42704c35bee42f08ccd6b340de1d66efaec7dca0/samples/ServerApplication/MyCustomProtocol.cs#L32

Is this expected? Because it only processes the packet received internally but I can't get the packet.

juliolitwin avatar May 22 '20 13:05 juliolitwin

In MyCustomProtocol in readasync, I can't get the received packet array, as in the example on the line:

Not sure what you mean...

davidfowl avatar May 23 '20 17:05 davidfowl