Released the WebRTC performance testing tool

[getroot]

OvenMediaEngine works in various server environments. The performance will also depend on the hardware/network performance of the server, as well as various variables such as input video quality and encoding profile. So we couldn't clearly answer the next question. "How many concurrent users can OvenMediaEngine accommodate?"

So we developed a tool to test the performance of your OvenMediaEngine on your own hardware environment, with the desired video input quality.

Please check the URL below. I look forward to your many advices. PR for improvement is also welcome (for example, a report that prints better)

https://airensoft.gitbook.io/ovenmediaengine/v/master/performance-tuning#getting-started-ovenrtctester

[getroot]

You should use the latest OvenMediaEngine from the master branch as your test target. The older OvenMediaEngine has a problem with the WebRTC RTP timestamp, so OvenRtcTester cannot calculate the video delay value. This has been patched in the latest master branch.

[getroot]

I was testing over 1000 sessions on AWS EC2 and found out that the ICE connectivity compatibility between OvenMediaEngine and pion/webrtc is not good. I improved this and re-released 0.12.4 with some bugs in the 0.12.3 release.

To test OvenMediaEngine with this performance test tool, please use OvenMediaEngine 0.12.4 or higher.

[getroot]

Today I tested the performance of OvenMediaEngine on AWS EC2. I've seen OvenMediaEngine reliably streaming 2.1Mbps video to 1600 concurrent users on a c5.4xlarge instance. To simulate 1600 concurrent users, 800 sessions were simulated using OvenRtcTester on two instances each, and playback was also confirmed in the browser of my PC.

The following is a screen captured during the test.

I started testing with a c5.4xlarge and Ubuntu20.04, and I'm experiencing a strange problem. For some unknown reason, network packets were lost (even though it was TCP), which caused a lot of problems. The server I have installed at my company is also Ubuntu 20.04, but there are no issues there. After trying various things, when I tested using a c5.4xlarge instance and Amazon Linux 2, all the problems went away. I recommend using Amazon Linux on AWS EC2.

Here is more information about the test environment.

[Output Stream information] Video : H.264, CBR, 2000Kbps, Keyframe Interval : 2, Profile : baseline Audio: OPUS, 128Kbps, Samplerate: 48000

[OvenMediaEngine server information] Instance Type: c5.4xlarge Number of instances: 1 OS: Amazon Linux OvenMediaEngine Version : master branch (added Amazon Linux support to prerequisites.sh today, otherwise same as 0.12.4) Setting: Modify only <Application><Publishers><StreamWorkerCount>16</StreamWorkerCount> in default Server.xml, (default 8) Clustering: No Origin-Edge, performance test with one Origin server (1 RTMP Input / 1600 WebRTC/tcp Output)

[OvenRtcTester server information] Instance Type: c5.4xlarge Number of instances: 2 (simulating 800 sessions each) OS: Amazon Linux Simulator: OvenRtcTester included in OvenMediaEngine 0.12.4 Simulator execution command: go run OvenRtcTester.go -url ws://host:3333/app/stream?transport=tcp -n 800

[Linux Tuning Settings]

[ec2-user@ip-172-31-56-213 ~]$ cat /etc/sysctl.conf
fs.file-max = 100000
net.core.somaxconn = 65535
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0

[ec2-user@ip-172-31-56-213 ~]$ cat /etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576

[getroot]

I found out today that Linux has a limit on the number of files that can be opened per service(process). There was also a limit to this in AWS' ubuntu, so socket connections were no longer possible at 1000 or more.

I made a commit by patching like below. OME is now capable of maximizing performance on most linux.

93c07c06aaaa088ea7e623a3a8980b9aa1297899

[bchah]

Hi @getroot I am really interested in testing this but I am having a problem with the pion/webrtc/v3 dependency. I can find a package for pion/webrtc but not v3, and the Go project refuses to build without it. Do you know the best way to import these dependencies?

Sorry for the amateur question! I am new to the Go environment (but I do remember playing the game with the black and white tiles a lot back in high school)

[getroot]

@bchah I've been studying "go" for the first time recently (less than a month :-)) to develop RtcTester. So I'm also not familiar with go yet. I've never seen one like you, especially. I've always worked fine with the go run OvenRtcTester.go command as stated in the manual. If it's for builds, could you try the go mod init command?

[fcqpl]

Hi, I can't get this tool to work.

Connection is in "Checking" state. After long time it goes to "Failed". Tested with ws:// tcp, udp and wss:// tcp, udp (with valid cert).

OvenMediaEngine v0.12.6 is started on [f9bf745e2ce5] (Linux x86_64 - 5.10.0-9-amd64, #1 SMP Debian 5.10.70-1 (2021-09-30)) OvenRtcTester version is latest from repo.

[2021-10-16 23:14:31.210] I [SPRtcSignalling:13] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f5424001130, #30, Connected, TCP, Nonblocking, 10.1.1.174:55312>
[2021-10-16 23:14:31.235] I [SPRtcSignalling:13] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/test_high on the WebRTC publisher. WebRTC(2)/Stream total(2)/App total(2)
[2021-10-16 23:14:41.909] I [SPRtcSignalling:13] WebRTC Publisher | webrtc_publisher.cpp:605  | Stop commnad received : #default#app/test_high/105
[2021-10-16 23:14:41.909] I [SPRtcSignalling:13] Monitor | stream_metrics.cpp:136  | A session has been stopped playing #default#app/test_high on the WebRTC publisher. Concurrent Viewers[WebRTC(1)/Stream total(1)/App total(1)]
[2021-10-16 23:14:41.909] I [SPRtcSignalling:13] Signalling | rtc_signalling_server.cpp:335  | Client is disconnected: <WebSocketClient: 0x7f542c053b20, <ClientSocket: 0x7f5424001130, #30, Disconnected, TCP, Nonblocking, 10.1.1.174:55312>> (#default#app / test_high, ufrag: local: fUQANt, remote: WyQVFDERdgoYCLoZ)
[2021-10-16 23:15:39.667] I [SPRtcSignalling:13] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f5424001130, #30, Connected, TCP, Nonblocking, 10.1.1.174:55318>
[2021-10-16 23:15:39.691] I [SPRtcSignalling:13] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/test_high on the WebRTC publisher. WebRTC(2)/Stream total(2)/App total(2)
[2021-10-16 23:15:52.254] I [SPRtcSignalling:13] WebRTC Publisher | webrtc_publisher.cpp:605  | Stop commnad received : #default#app/test_high/106
[2021-10-16 23:15:52.254] I [SPRtcSignalling:13] Monitor | stream_metrics.cpp:136  | A session has been stopped playing #default#app/test_high on the WebRTC publisher. Concurrent Viewers[WebRTC(1)/Stream total(1)/App total(1)]
[2021-10-16 23:15:52.254] I [SPRtcSignalling:13] Signalling | rtc_signalling_server.cpp:335  | Client is disconnected: <WebSocketClient: 0x7f542c053420, <ClientSocket: 0x7f5424001130, #30, Disconnected, TCP, Nonblocking, 10.1.1.174:55318>> (#default#app / test_high, ufrag: local: bl2I3c, remote: bibUDPFDMXkMdhjn)

root@ome-stress:/opt/OvenMediaEngine/misc/oven_rtc_tester# go run OvenRtcTester.go -url wss://example.com:3334/app/test_high?transport=tcp
client_0 connection state has changed checking
client_0 has started
<Summary>
Running time : 2562047h47m16.854775807s
Number of clients : 1
ICE Connection State : New(0), Checking(1) Connected(0) Completed(0) Disconnected(0) Failed(0) Closed(0)
<Summary>
Running time : 2562047h47m16.854775807s
Number of clients : 1
ICE Connection State : New(0), Checking(1) Connected(0) Completed(0) Disconnected(0) Failed(0) Closed(0)
^CTest stopped by user
***************************
Reports
***************************
<Summary>
Running time : 2562047h47m16.854775807s
Number of clients : 1
ICE Connection State : New(0), Checking(1) Connected(0) Completed(0) Disconnected(0) Failed(0) Closed(0)
<Details>
[client_0]
        running_time(2562047h47m16.854775807s) connection_state(checking) total_packets(0) packet_loss(0)
        last_video_delay (0.0 ms) last_audio_delay (0.0 ms)
        total_bytes(0bytes) avg_bps(0bps) min_bps(0bps) max_bps(0bps)
        total_video_frames(0) avg_fps(0.00) min_fps(0.00) max_fps(0.00)

client_0 has stopped
root@ome-stress:/opt/OvenMediaEngine/misc/oven_rtc_tester#

root@ome-stress:/opt/OvenMediaEngine/misc/oven_rtc_tester# go run OvenRtcTester.go -url wss://example.com:3334/app/test_high
client_0 connection state has changed checking
client_0 has started
udp4 host 10.1.1.174:36632 candidate found
<Summary>
Running time : 2562047h47m16.854775807s
Number of clients : 1
ICE Connection State : New(0), Checking(1) Connected(0) Completed(0) Disconnected(0) Failed(0) Closed(0)
<Summary>
Running time : 2562047h47m16.854775807s
Number of clients : 1
ICE Connection State : New(0), Checking(1) Connected(0) Completed(0) Disconnected(0) Failed(0) Closed(0)
^CTest stopped by user
***************************
Reports
***************************
<Summary>
Running time : 2562047h47m16.854775807s
Number of clients : 1
ICE Connection State : New(0), Checking(1) Connected(0) Completed(0) Disconnected(0) Failed(0) Closed(0)
<Details>
[client_0]
        running_time(2562047h47m16.854775807s) connection_state(checking) total_packets(0) packet_loss(0)
        last_video_delay (0.0 ms) last_audio_delay (0.0 ms)
        total_bytes(0bytes) avg_bps(0bps) min_bps(0bps) max_bps(0bps)
        total_video_frames(0) avg_fps(0.00) min_fps(0.00) max_fps(0.00)

client_0 has stopped
client_0 connection state has changed closed
root@ome-stress:/opt/OvenMediaEngine/misc/oven_rtc_tester#

[getroot]

@fcqpl This is a problem where OvenRtcTester cannot connect to the IceCandidate and TcpRelay servers you set in OvenMediaEngine. If you set it to *, can you try setting it to an IP that OvenRtcTester can connect to?

[fcqpl]

If you set it to *, can you try setting it to an IP that OvenRtcTester can connect to?

Yes, after change to: <IceCandidate>10.1.1.175:10000/udp</IceCandidate> It's working. Thanks!

[harinandans]

Is there a way to create several webrtc inputs? I have a use case of thousands of webrtc inputs and each having just one output (mostly HLS) to test load.

[htliu6]

@getroot Hello! Several days no see! I wonder how the video delay is calculated in OvenRtcTester, could you please give an introduction to this?