libdatachannel icon indicating copy to clipboard operation
libdatachannel copied to clipboard

Published 20 hours ago • verified publisher icon paullouisageneau

benchmark on Windows

Open murat-dogan opened this issue 4 years ago • 15 comments

This is result of benchmark on windows

Description 1: v=0
o=- 2649482864 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:data
a=sctp-port:5000
a=max-message-size:262144
a=ice-options:trickle
a=ice-ufrag:Sb7x
a=ice-pwd:J7k0Pslc2jT2aJIDtGqijR
a=setup:actpass
a=tls-id:1
a=fingerprint:sha-256 AB:56:43:64:E8:F5:49:34:35:93:A2:73:BF:5D:34:2D:75:A5:52:5D:BA:6F:51:CF:EE:15:FB:78:D0:C2:9A:A2

Description 2: v=0
o=- 2642587225 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:data
a=sctp-port:5000
a=max-message-size:262144
a=ice-options:trickle
a=ice-ufrag:1x2Z
a=ice-pwd:fkQnpmrsqsbh/s/ZtWIhoo
a=setup:active
a=tls-id:1
a=fingerprint:sha-256 AB:56:43:64:E8:F5:49:34:35:93:A2:73:BF:5D:34:2D:75:A5:52:5D:BA:6F:51:CF:EE:15:FB:78:D0:C2:9A:A2

Gathering state 2: in_progress
Candidate 2: a=candidate:1 1 UDP 2015363327 10.0.0.12 18703 typ host
Candidate 2: a=candidate:2 1 UDP 2015363583 fe80::b415:a528:ee00:3900 50283 typ host
Gathering state 2: complete
Gathering state 1: in_progress
State 1: State 2: connecting
connecting
Candidate 1: a=candidate:1 1 UDP 2015363327 10.0.0.12 15625 typ host
Candidate 1: a=candidate:2 1 UDP 2015363583 fe80::b415:a528:ee00:3900 14642 typ host
Gathering state 1: complete
State 2: connected
State 1: connected
DataChannel open, sending data...
Received: 14286 KB
Received: 21954 KB
Received: 57605 KB
Received: 89062 KB
Received: 128317 KB
Received: 147388 KB
Received: 186053 KB
Received: 240054 KB
Received: 283176 KB
Received: 304475 KB
Test duration: 30000 ms
Connect duration: 231 ms
Goodput: 10.01 MB/s (80.08 Mbit/s)
State 1: closed
State 2: closed

On linux it is typically 500MBit/s.

Could you please check on your side?

murat-dogan avatar Jun 24 '20 06:06 murat-dogan

Above results was on debug build. This is on release;

Description 1: v=0
o=- 3716173587 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:data
a=sctp-port:5000
a=max-message-size:262144
a=ice-options:trickle
a=ice-ufrag:FzX8
a=ice-pwd:1oa9VCePXY5Nrh6oeIXotu
a=setup:actpass
a=tls-id:1
a=fingerprint:sha-256 2F:C7:48:14:75:DF:E4:01:8B:8D:C8:27:C0:F7:FD:F9:75:05:5C:A8:84:FF:2E:91:EB:C0:ED:B3:5F:8A:9F:50

Description 2: v=0
o=- 1574102576 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:data
a=sctp-port:5000
a=max-message-size:262144
a=ice-options:trickle
a=ice-ufrag:7M9A
a=ice-pwd:TAlId3ag26a7vCZZWxamuk
a=setup:active
a=tls-id:1
a=fingerprint:sha-256 2F:C7:48:14:75:DF:E4:01:8B:8D:C8:27:C0:F7:FD:F9:75:05:5C:A8:84:FF:2E:91:EB:C0:ED:B3:5F:8A:9F:50

Gathering state 2: in_progress
Candidate 2: a=candidate:1 1 UDP 2015363327 10.0.0.12 4316 typ host
Candidate 2: a=candidate:2 1 UDP 2015363583 fe80::b415:a528:ee00:3900 23498 typ host
Gathering state 2: complete
Gathering state 1: in_progress
State 1: State 2: connecting
connecting
Candidate 1: a=candidate:1 1 UDP 2015363327 10.0.0.12 39683 typ host
Candidate 1: a=candidate:2 1 UDP 2015363583 fe80::b415:a528:ee00:3900 32806 typ host
Gathering state 1: complete
State 2: connectedState 1: connected

DataChannel open, sending data...
Received: 95353 KB
Received: 177599 KB
Received: 295759 KB
Received: 410249 KB
Received: 433120 KB
Received: 543481 KB
Received: 607968 KB
Received: 671537 KB
Received: 680712 KB
Received: 774885 KB
Test duration: 30000 ms
Connect duration: 291 ms
Goodput: 25.795 MB/s (206.36 Mbit/s)
State 1: closed
State 2: closed

murat-dogan avatar Jun 24 '20 07:06 murat-dogan

There was a network error. I updated with new results.

Outputs of client app from benchmark branch. Release build, on different Windows PCs, 100MBit Network

Offerer

The local ID is: TmUy
Waiting for signaling to be connected...
WebSocket connected, signaling ready
Offering to djTf
Creating DataChannel with label "benchmark"
Gathering State: in_progress
Gathering State: complete
State: connecting
State: connected
DataChannel from djTf open, sending data...
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
State: closed

Answerer

The local ID is: djTf
Waiting for signaling to be connected...
WebSocket connected, signaling ready
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Goodput: 0 MB/s (0 Mbit/s)
Answering to TmUy
Goodput: 0 MB/s (0 Mbit/s)
Gathering State: in_progress
State: connecting
Gathering State: complete
State: connected
DataChannel from TmUy received with label "benchmark"
Goodput: 3.382 MB/s (27.056 Mbit/s)
Goodput: 10.669 MB/s (85.352 Mbit/s)
Goodput: 9.817 MB/s (78.536 Mbit/s)
Goodput: 10.839 MB/s (86.712 Mbit/s)
Goodput: 10.828 MB/s (86.624 Mbit/s)
Goodput: 10.865 MB/s (86.92 Mbit/s)
Goodput: 10.88 MB/s (87.04 Mbit/s)
Goodput: 10.773 MB/s (86.184 Mbit/s)
Goodput: 10.88 MB/s (87.04 Mbit/s)
Goodput: 10.852 MB/s (86.816 Mbit/s)
Goodput: 10.854 MB/s (86.832 Mbit/s)
Goodput: 10.878 MB/s (87.024 Mbit/s)
Goodput: 10.828 MB/s (86.624 Mbit/s)
Goodput: 10.867 MB/s (86.936 Mbit/s)
Goodput: 10.867 MB/s (86.936 Mbit/s)
Goodput: 10.865 MB/s (86.92 Mbit/s)
Goodput: 10.865 MB/s (86.92 Mbit/s)
Goodput: 10.815 MB/s (86.52 Mbit/s)
Goodput: 10.878 MB/s (87.024 Mbit/s)
Goodput: 10.839 MB/s (86.712 Mbit/s)
Goodput: 10.564 MB/s (84.512 Mbit/s)
Goodput: 8.178 MB/s (65.424 Mbit/s)

murat-dogan avatar Jun 24 '20 07:06 murat-dogan

So Only thing that take my attention is benchmark result of windows. On linux it is 500MBit/s On Windows it is 200MBit/s

murat-dogan avatar Jun 24 '20 08:06 murat-dogan

The 100Mbps LAN result is really good since the protocol stack introduces ~10% overhead. I'm not surprised by the lower performance locally, I guess the limitation comes from inoptimal code for Windows somewhere since everything in the stack is basically for Unix-like platforms and ported to Windows. Is it a 32 or 64bits build? The usual suspect is libnice, could you try switching to libjuice to compare?

paullouisageneau avatar Jun 24 '20 09:06 paullouisageneau

There is a compile error on line;

https://github.com/paullouisageneau/libjuice/blob/7cd5e53d54f83048122e14a6cc52f6ac38b7a98c/src/udp.c#L134

For windows EADDRINUSE should be WSAEADDRINUSE

murat-dogan avatar Jun 24 '20 09:06 murat-dogan

With libjuice (32 Bit, libnice was 32 Bit also);

0
o=- 72248827 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:data
a=sctp-port:5000
a=max-message-size:262144
a=ice-options:trickle
a=ice-ufrag:bPLr
a=ice-pwd:KPheV0Dd+3r70tTD9C46E/
a=setup:actpass
a=tls-id:1
a=fingerprint:sha-256 74:ED:20:0C:DE:5B:A1:AB:84:A3:ED:1C:2D:C7:44:A3:8A:B3:D5:FB:F4:7B:8E:A1:A8:32:CF:36:FB:0D:22:C6

Description 2: v=0
o=- 1225685164 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:data
a=sctp-port:5000
a=max-message-size:262144
a=ice-options:trickle
a=ice-ufrag:orDp
a=ice-pwd:K+JJE2G9PawAi05miwXSe/
a=setup:active
a=tls-id:1
a=fingerprint:sha-256 74:ED:20:0C:DE:5B:A1:AB:84:A3:ED:1C:2D:C7:44:A3:8A:B3:D5:FB:F4:7B:8E:A1:A8:32:CF:36:FB:0D:22:C6

Gathering state 2: in_progress
Candidate 2: a=candidate:1 1 UDP 2122317823 10.0.0.12 59212 typ host
Gathering state 1: State 2: in_progressconnecting

Candidate 1: a=candidate:1 1 UDP 2122317823 10.0.0.12 59213 typ host
Gathering state 2: complete
State 1: connecting
Gathering state 1: complete
State 2: connected
State 1: connected
DataChannel open, sending data...
Received: 18808 KB
Received: 66321 KB
Received: 100596 KB
Received: 135067 KB
Received: 164820 KB
Received: 203355 KB
Received: 233370 KB
Received: 266006 KB
Received: 318237 KB
Received: 326036 KB
Test duration: 30000 ms
Connect duration: 327 ms
Goodput: 10.831 MB/s (86.648 Mbit/s)
State 1: closed
State 2: closed

murat-dogan avatar Jun 24 '20 09:06 murat-dogan

The EADDRINUSE should be fixed by https://github.com/paullouisageneau/libjuice/commit/7fa9ff2fd4ad7b89b86e4f9915dcc49cf896d9b0

Yeah actually I just remembered the C11 support is lacking in MSVC so atomics in particular are disabled: https://github.com/paullouisageneau/libjuice/blob/master/CMakeLists.txt#L9 This forces the send function to lock a mutex each time instead which is extremely slow. I should so something about that in the future, but in the meantime we can't use it for comparison.

When running with libnice does the program use a full processor core?

paullouisageneau avatar Jun 24 '20 11:06 paullouisageneau

Please not that I am using mingw as compiler.

benchmark is not using full processor core, let's say ~70% roughly.

murat-dogan avatar Jun 24 '20 12:06 murat-dogan

It seems C11 atomics are also missing from mingw: https://mingw-w64.org/doku.php/contribute#c11_threading_and_atomics

The fact that it doesn't cap out to 100% suggests the bottleneck is not processing but something like a buffer size somewhere. Do you use Windows 10?

paullouisageneau avatar Jun 24 '20 13:06 paullouisageneau

Win 10

murat-dogan avatar Jun 25 '20 05:06 murat-dogan

How to define Test duration: 30000 ms and Connect duration: 313 ms ?

alicera avatar Aug 12 '20 07:08 alicera

What do you mean?

paullouisageneau avatar Aug 12 '20 09:08 paullouisageneau

For example, Test duration : it represent the time of the constant 30s, (Meaningless). Connect duration: it represent the time of "send SDP" + Candidate . Goodput: DataChannel open, sending data speed (P2P connect).

image

In fact, I dont know about it

alicera avatar Aug 12 '20 10:08 alicera

Oh, you mean the meaning of measured values.

Test duration: This is constant and set in the test. You can get a more reliable measure by increasing it.

Connect duration: Time between the peer connection creation and the P2P actually being connected. Goodput: Average throughput of application data in the Data Channel. The network throughput will be a bit higher due to the protocol overhead.

paullouisageneau avatar Aug 12 '20 11:08 paullouisageneau

For the record, performance will now be better with libjuice with the logging optimization in https://github.com/paullouisageneau/libdatachannel/pull/179

paullouisageneau avatar Sep 23 '20 14:09 paullouisageneau