LiteNetLib
LiteNetLib copied to clipboard
RevenantX
Implement sliding window scale logic.
I have been testing LiteNetLib, Lidgren-gen3 and Lidgren-old (the one from Google Code) and Lidgren-old is beating the competitors by far... On my test I have 750 clients connected to a single server instance, receiving ~15 reliable msgs/s and ~100 unreliable msgs/s and Lidgren-old can handle all this data perfectly! Using the same "game code" but using Lidgren-gen3 or LiteNetLib the test "fails" between 100 and 120 clients... Maybe I am missing something with LiteNetLib API, so I can achieve better performance... Do you have any idea where I am doing wrong with your Lib? If you need I can share the code :)
- My test is using my area of interest management layer, so it can efficiently send to clients only data important to them. Only using this technique I was able to achieve 750 clients on a single server :) Every client is also sending ~15 unreliable msgs/s (movement messages)
Thanks and keep the awesome work!
@lzaiats Hi!
? If you need I can share the code :)
Can you show minimal sample code that works slower than Lidgren-old?
Basically it is UDP congestion the problem.. Lidgren-old appears to drop less packets than LiteNetLib, for example... I will reproduce a little sample code, with LiteNetLib, Lidgren-gen3 and Lidgren-old so you can see the "thing" on your local machine ;)
BTW, i've implemented a custom NetSerializer that use MessagePack (the optimized from neuecc) for serialization and it works great! Maybe it can be usefull to other LiteNetLib users ;)
Hi @RevenantX !
Here is the simple benchmark I made... It sends reliable and unreliable messages and count sent and received messages... LiteNetLib reliable performance is not good compared to Lidgren-old! On my "game project" after some time, the clients simply stops receiving reliable instantiate messages and the game stops working... Another important point to mention is the memory consumption... Lidgren-old test is almost not using memory allocations at all, while LiteNetLib test uses > 2Gig...
Here is the results:
Testing Lidgren-old... Processing... CLIENT SENT -> Reliable: 261800, Unreliable: 748000 SERVER RECEIVED -> Reliable: 261800, Unreliable: 748000
Testing LiteNetLib... Processing... CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 51215, Unreliable: 750000
@lzaiats thank you! I will investigate this. Something wrong with send speed.
@lzaiats high memory consumption caused by packet pooling. I think i must add some limits to pooled packets.
By staring a second client on the same machine I was able to double the bandwidth transfer on the same connection so this suggests a rate limit inside each client. Would a bounty help you find the time to work on this issue?
@mgamache i know why this problem exits) As i said before - this because i don't implemented "sliding window scale" logic. This requires some time to implement.
Would a bounty help you find the time to work on this issue?
Donations always help to find the time to work on library entirely :) But i can't promise fast fix(implementaiton) right now, because i'm working on two projects. But soon (end of december) i will have some time for library features and improvements.
Hey, remember me?
So was going to ask if you want to share profits on releasing https://github.com/Saishy/TinyBirdNet-Unity at the asset store. I was intending it to be free for free projects and paid for commercial ones. (still open source tho)
@lzaiats hi! Try latest version from github. I heavily optimized sending code. Your problem (with 750 clients) must be fixed. (but not benchmark)
@Saishy
Hey, remember me?
Yes)
So was going to ask if you want to share profits on releasing https://github.com/Saishy/TinyBirdNet-Unity at the asset store. I was intending it to be free for free projects and paid for commercial ones. (still open source tho)
You can donate any amount that you think is necessary)
@RevenantX Heeeey! Awesome news!
I ran the tests changing the values of the NetConstants.DefaultWindowSize and here are the results:
---- WindowSize = 16 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 11984, Unreliable: 749242
---- WindowSize = 32 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 23968, Unreliable: 749236
---- WindowSize = 64 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 46080, Unreliable: 749177
---- WindowSize = 128 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 88832, Unreliable: 749097
---- WindowSize = 256 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 137728, Unreliable: 749429
---- WindowSize = 512 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 219674, Unreliable: 749907
---- WindowSize = 1024 Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 SERVER RECEIVED -> Reliable: 262150, Unreliable: 750000
IMPRESSIVE!
I'll do some more testing using the real 750 clients and share the results with you! Thank you!
@lzaiats don't increase it manually. It must be increased automatically (later). My current fix must increase performance of real game server (with many clients!). (not file transfer to 1 peer)
@RevenantX ;) Actually I only increased the value by hand to understand the maximum throughput the lib could achieve... On the game (production) I will not touch this number ;)
One thing that is still making me a little confused is that on all circunstances, the bytes sent by the client and the bytes received by the server are almost the same... But the number of messages are completely different... Have a look:
Testing LiteNetLib... Processing... DataSize: 43537500b, 42517kb, 41mb CLIENT SENT -> Reliable: 262500, Unreliable: 750000 CLIENT STATS: BytesReceived: 36632 PacketsReceived: 1599 BytesSent: 41320993 PacketsSent: 839281 PacketLoss: 2797 PacketLossPercent: 0
SERVER RECEIVED -> Reliable: 89536, Unreliable: 749593 SERVER STATS: BytesReceived: 41335938 PacketsReceived: 839586 BytesSent: 36632 PacketsSent: 1599 PacketLoss: 0 PacketLossPercent: 0
@lzaiats after some time all reliable messages will be received. Add more time before showing statistics) They just in queue.
@lzaiats
I'll do some more testing using the real 750 clients and share the results with you! Thank you!
Any news?)
@Izaiats Hi,
Did you try with a lower DefaultUpdateTime ? Like 5 ms. Can it help ? Maybe the thread shouldnt stop until there is nothing else to process, wait after that.
@nxrighthere Maybe 1 ms sleep too much. Maybe it needs to none stop update until its done.
@DanisJoyal I tried a lot of configurations and I had no luck!
@nxrighthere Nice to see your benchmark suite! Also I made one using exact the same libs you did and the results are just the same! Using ENet wrapper gives the best result always! If you want to see good results with another lib, try this OLD lidgren version: https://code.google.com/archive/p/lidgren-network/downloads , the one from Jan 22, 2010... It's almost the same API as Lidgren-v3 but will show you results comparable to ENet ;)
@RevenantX I stopped benchmarking for now, maybe we can continue the investigation using @nxrighthere test suite now, since it's more complete and modular than mine!
@lzaiats Hi,
I just ran your UDPLibBench.zip. I ve got to adapt it to my latest version, on my fork. So far, Ive got this:
Testing LiteNetLib... Processing... CLIENT SENT -> Reliable: 262500, Unreliable: 750000 PacketLossPercent: 0(672) Bytes Received/Sent: 57281/50915910 Packets Received/Sent: 4215/1007890 SERVER RECEIVED -> Reliable: 262500, Unreliable: 750000 PacketLossPercent: 0(0) Bytes Received/Sent: 51156422/57568 Packets Received/Sent: 1012621/4241
No Issue, except a little bit packets loss on client side.
Im running 2 clients thread at the same time on the same server to see if it last.
Cheers
Oups! The result just came out of the oven :)
Testing LiteNetLib... Processing... CLIENT SENT -> Reliable: 262500, Unreliable: 750000 PacketLossPercent: 0(1120) Bytes Received/Sent: 58801/51260444 Packets Received/Sent: 4343/1014618 CLIENT SENT -> Reliable: 262500, Unreliable: 750000 PacketLossPercent: 0(1078) Bytes Received/Sent: 58805/51279396 Packets Received/Sent: 4344/1014994 SERVER RECEIVED -> Reliable: 525000, Unreliable: 1500000 PacketLossPercent: 0(0) Bytes Received/Sent: 102643216/117768 Packets Received/Sent: 2031599/8704
The DefaultWindowSize = 64;
- Adding My CPU was at 100% on a i5-2700k with 2 clients
@Izaiats
I left the UpdateTime = 15ms. But my KcpChannel overwrite this value to 10ms. But, I added a "skip sleep thread" if there was a packet sent during the Update. Maybe I can try without it.
Wow! ok !
UpdateTime = 1ms without my "Skip sleep thread"
Testing LiteNetLib... Processing... CLIENT SENT -> Reliable: 262500, Unreliable: 750000 PacketLossPercent: 0(1210) Bytes Received/Sent: 12226/38550732 Packets Received/Sent: 429/770182 CLIENT SENT -> Reliable: 262500, Unreliable: 750000 PacketLossPercent: 0(1268) Bytes Received/Sent: 12406/38563786 Packets Received/Sent: 444/770435 SERVER RECEIVED -> Reliable: 51521, Unreliable: 1435184 PacketLossPercent: 0(0) Bytes Received/Sent: 75644322/40118 Packets Received/Sent: 1509862/1437
Something strange about the packet loss and everything ! Server just receive 51521 on 525000. Only 1210 + 1268 packet loss ?!
And my CPU was at 70-75% instead.
@izaiats, @nxrighthere
I ran multiple times, with different settings. With the "skip sleep thread", it works perfectly.
If I use both UpdateTime = 1ms and "skip sleep thread", Im getting more packet loss from clients, like 27000 for both instead of 1000. But, the receive counts are good even with those settings.
Im done here :) Have fun.
@DanisJoyal I tried to run the test with your fork. Everything goes well, as long as only 1 client is connected to the server. As soon as 2 or more clients are connected and start sending the messages, CPU usage increases to 80%-100% and my system becomes unstable.
@nxrighthere Well, Im trying to run performance profiler to optimize. It seems that SendNextPacket on the reliable channel is taking 50% of the time vs unreliable 8%. It looks like a Lock on _pendingPacket list
Well, I guess that Im wrong. ReliableChannel.SendNextPackets is always returning true. So the CPU is chocking, slowing down the clients. its suppose to take 10 sec to run. It takes 2 min instead.
Sorry guys
@nxrighthere Hi, can you try again with my latest fixes ? Its working on my side without using 100% cpu. In fact, it turns around 50% now. Thx
@DanisJoyal It works better, but CPU usage is still very high compared to the master branch:
64 clients 32% vs 2%
500 clients 70% (160 was connected - overload) vs 60% (500 was connected - success)
@nxrighthere Ok, but, are you sure that the reliable channel is working perfectly ? I added a sequence check in the data sended. It failed. It sent the good number of msg, but not in the good order. Maybe its just my fork ... Im a little bit confuse ! Because the way to test it influence the result.
With your fork, reliable channel as well as unreliable can't handle it. Here is the results of the master branch.
@nxrighthere The real question is: How are you testing 500 clients ? Because it did my fix are based on the UDPLibBench.zip. And Im not sure that its a good way to test the lib. Can you share your code ?
Thank a lot.
@DanisJoyal @nxrighthere found some places that i can optimize. So i will upload improved version soon. But i see that most CPU usage in socket.send :)
I noticed this CPU use as well.
Right now on my stress tool, websockets is out performing UDP on more than 200 connections.
@nxrighthere @NVentimiglia this bench is not fully correct. Server can handle many connections. If that 500 or more clients on separate machine) This CPU usage is just because 500 clients logic slowing down whole system.
@RevenantX
You are right, but its showing how much ressources are you taking. For example, if Enet or lidgren can do the test but not litenetlib, there is an issue in the runtime of litenetlib. Its just an example. No ?
So far, Enet is unbeatable. But, in pure C#, liteNetLib is the best from the tests I did.
01-07-2018
-MOCK- Connected : 100 Requests Per Second : 1,652,450.17 Latency : 0.00 ms
-UDP- Connected : 8 Requests Per Second : 2,856.17 Latency : 2.27 ms
Connected : 100 Requests Per Second : 978.02 Latency : 101.06 ms
-WS- Connected : 100 Requests Per Second : 3,660.91 Latency : 59.17 ms
Side note, I have a high level library with an agnostic transport layer. I can scale WS and Mock connections to 5k connections without much slow down. I have tried my UDP connection using my own async threading and the internal que with little variation between the two. My implementation is pretty close to the example implementation. The only real difference is the fact that I serialize and deserialize. I dont think its my code, as the mock and WS go through this same system. Moreover, I have replicated this problem without my high level library. The test is a simple echo test.
@DanisJoyal Lidgren and LiteNetLib passed and failed same tests. Larger memory consumption is because of larger pool (you can check this by setting PoolSize to lower values). Enet can handle because of congestion control and this is "C" library that obviously faster :)
Everything is fine with the benchmark, the problem in the library. @DanisJoyal has successfully fixed some things and here's the result with 2000 clients.
@RevenantX
I did some optimization. The array: _peers was blocking the flow mostly. I just did a pull request about it. On my PC, I can run almost 2000 clients at the same time.
But, It should take 2 mins to complete the test. So, its still taking too much ressources.
@DanisJoyal @nxrighthere i merged changes into separate branch and tested. And on local machine i don't see any improvements O_o. Do you use release version of library and becnh? or debug?
@RevenantX
You need to push it at the limit. If you re telling me that both fail at the same time, ok ... its strange. Because I ran tests during 4-5 hrs and I saw an improvement.
@DanisJoyal no, there no failes. But i tested 200 clients and got 70-100% cpu usage just by socket.send() (i5-4690K 4.2 GHz) in both tests.
@RevenantX There is a huge difference running the application via Visual Studio or by launching the application from explorer. Even if you are in release. Or, you may run without debugging. CTRL-F5
@nxrighthere @DanisJoyal received better results withou merge. As separate application. Clients failed both on 714. But CPU usage was lower before fixes. @nxrighthere can you please check on you machine both branches(https://github.com/RevenantX/LiteNetLib/tree/master and https://github.com/RevenantX/LiteNetLib/tree/optimisation) and test?
@RevenantX Wow! It really surprises me. Im testing with your new reliable optimisation. Maybe CPU usage its higher on mine. I have improve that, but it failed at 714 ??? ... very strange. I didnt trick the screenshot !! Anyway ... thats fine.
@RevenantX I know. The ??? means that Im very confused about your results. I dont know what to think.
Well, the master branch: 652 started, 638 connected, 0 dropped (CPU usage 57%). Everything going well, but after some time the application is crashed with the error:
System.NullReferenceException: in LiteNetLib.NetPeer.SendRawData(NetPacket packet) in LiteNetLib.ReliableChannel.SendNextPackets()) in LiteNetLib.NetPeer.Flush() in LiteNetLib.NetPeer.Update(Int32 deltaTime) in LiteNetLib.NetManager.UpdateLogic() in System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) in System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) in System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) in System.Threading.ThreadHelper.ThreadStart() System.NullReferenceException: Ссылка на объект не указыinает на экземпляр объекта. in LiteNetLib.NetPeer.SendRawData(NetPacket packet) in LiteNetLib.ReliableChannel.SendNextPackets() in LiteNetLib.NetPeer.Flush() in LiteNetLib.NetPeer.Update(Int32 deltaTime) in LiteNetLib.NetManager.UpdateLogic() in System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) in System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
The second branch: 1000 started, 1000 connected, 0 dropped (CPU usage 87%). A test has been completed in 6 minutes (must be completed in 2 minutes), but it's passed with 99.9% result. At the end of the process, almost the all clients are dropped for some reason.
@nxrighthere Previously, you wasnt able to reach 1000 clients ? Now, you can ?
@nxrighthere yeah there is bug (very rare o_o) in reliable channel after optimisation. I will try fix.
@RevenantX Do we have access to your application test ? Seriously, the way to test does all the difference. From, @lzaiats UDPCLientBench to benchmarkNet .... there is a huge difference.
@RevenantX Ok. But, I didnt use the source code. Im using the binary application. Im just replacing the litenetlib.dll in the folder. Otherwise, be sure that its compiling in x64.
After 900 connections, the clients are starts dropping and the process slowly freezes.
@nxrighthere @DanisJoyal checked both versions after fix. Got almost same results in both cases (1000 clients work without drops 50-58% cpu usage).
@nxrighthere @DanisJoyal better will be continue this conversation in some messenger (skype, telegram, etc.)
It looks like the search in Skype is broken right now... I unable to find you.
@nxrighthere i will partially implement your optimizations. And compare to all optimizations later.
@nxrighthere @RevenantX you can use mattermost.
https://about.mattermost.com/download/#mattermostApps
the url server is https://demo.mattermost.com/
The room or channel is https://demo.mattermost.com/space-motors/channels/litenetlib
Hi, sorry, I changed the room name:
https://demo.mattermost.com/space-motors/channels/litenetlib-discussion
@RevenantX Hi, I was comparing Enet with C# SendTo. Enet uses WSASendTo that it does buffer management in none blocking mode. You may send 1K byte and it will immediately return 1K sent even if its not sent. It will send it later.
C# Use UnsafeNclNativeMethods.OSSOCK.sendto instead of UnsafeNclNativeMethods.OSSOCK.WSASendTo.
Its unfair :) Ive tried in none blocking mode. Doesnt change anything except that SendTo is faster, but ReceiveFrom is sucking. ReceiveFrom is none blocking too ... always have to check.
Cheers
https://github.com/Microsoft/referencesource/blob/master/System/net/System/Net/Sockets/Socket.cs
http://labs.developerfusion.co.uk/SourceViewer/browse.aspx?assembly=SSCLI&namespace=System.Net&type=UnsafeNclNativeMethods
@DanisJoyal this will work for windows (for now without implementation for linux, osx, android, etc.). I've added winsock usage. So you can test this by selecting WIN32_UNSAFE build type in LiteNetLib project.
@DanisJoyal @nxrighthere added ReleseUnsafeWin32 version. And fixed reliable channel. So now just release version almost hit 1000 clients (~990). And ReleaseUnsafeWin32 (with just native sendto, receivefrom) passes test. @DanisJoyal so don't optimize not bottleneck places) This is just socket calls)
Okay, unsafe version is working much better! But, after 700 connections CPU usage raises 92% and the whole process becomes unresponsive. I monitored what's was happening and I found that more than 60% of the time is spent in GC and my CPU just can't handle it. Generation zero grows dramatically. With enabled Sustained Low Latency, it works better, but GC is still killing the performance.
@nxrighthere
Compared to ENet time in GC is 5%.
C library -> GC :) But this is strange anyway. I slightly reduced possible place in "receive" method.
Works better, now 750 connections is reached. The overall CPU usage and the generation zero are more stable. No drops by the way. The result with 700 clients. The old version can't complete the test with more than 550 clients. Very nice progress!
I'm so confused right now... I did a test with the old version of Lidgren as @lzaiats suggested, and you know what? It's passed the test with 1000 clients... CPU usage 60%, memory consumption 360 megabytes.
@nxrighthere Bingo! :) The old lidgren version is far better than the gen3, and its also almost the same performance than ENet... Amazing ;)
Thats funny. So, you re telling me that google are so dumb to create a new version of the lib worst than the first one. I think that it s an issue from the .Net Framework. Maybe you can try the old Lidgren to use .Net Framework > 4.5. I didnt have time since 1 week to continue my investigation. But, I saw that the framework its not using the same sendto function as Enet. I will check that this week end. Cheers.
You re overwriting the framework set by default ? The old lidgren is using .Net Framework 2.0. Anyway, if Lidgren is able to do it. LiteNetLib should be able too.
I'm not using the csproj at all, I compile everything directly with Roslyn.
@nxrighthere @DanisJoyal added DISABLE_IPV6 ifdef. So you can disable ipv6 thread (so this is 1000 less threads in that test). Passed test with 1200 clients.
Yea, works better now. 900 clients are connected and after that the CPU usage raises 92% and the application becomes unresponsive. But very nice anyway.
GC is still the biggest performance killer. I tried to build the library with built-in Roslyn's optimization logic and now 950 connections is reached, but the server thread just crashed.
https://github.com/DanisJoyal/LiteNetLib/commit/69e8244fd48076b52e1503ad9cf7b624e8c31ce5 https://github.com/DanisJoyal/LiteNetLib/commit/104fbf20b31d1f394e27c1ba9dc74a1feae1d371