[bug] Livekit's CPU usage

[furkanKotic]

Describe the bug When I test my application on Flutter iPhone, livekit's CPU usage is as follows:

Viewer CPU usage when there is a streamer in the room and his cameras are off:

Viewer CPU usage when there is a streamer in the room and his cameras are on:

Viewer CPU usage when there are two streamers in the room and their cameras are on:

Viewer CPU usage when there are three streamers in the room and their cameras are off:

Viewer CPU usage when two streamers in the room with their cameras on and one streamer with his camera off:

When I test without a livekit connection in the same application, I observe 6% CPU usage. In this case, we observe 50-60% CPU usage per broadcaster in Livekit. These processor uses cause the phone to overheat and increase battery usage. I think Livekit needs optimization.

Platform information

• Flutter version: Flutter 3.13.4
• Plugin version: livekit_client: 1.3.5
• Flutter target OS: iOS

[Parfyonator]

Hi. Users of my app that has video call integrated by means of this plugin also experience overheating. For some of them it is moderate and for the others it is pretty high.

[cloudwebrtc]

hey @furkanKotic @Parfyonator, Hey, can you guys confirm what video code was being used when this issue occurred, and what phone model you were using? I think the video code that consumes the most power in libwebrtc is usually the video codec.

[Parfyonator]

Hi @cloudwebrtc. In my case it is 14 pro max. One of the users reported that after 30 min of call it became so hot that the app became very laggy due to the very high temperature of the phone.

[furkanKotic]

Hi @cloudwebrtc. The scenario of @Parfyonator is also valid for me. My users report that the phone overheats after 30 minutes, the app becomes extremely slow, and the phone gives a high temperature warning. We are using H264 codec. We know that it is available in all recent iPhone models.

Some tested models: iPhone 15 pro max iPhone 14 pro max iPhone 14 iPhone X

I have all of these devices. These devices experience overheating when broadcasting or just participating as a viewer. But there is no problem on Android devices.

Resolution: h540_43 Bitrate: 1000 Framerate: 15

[dragontran]

Heyo! I wanted to point out that I've observed high CPU usage and high temps during extended use in my project, the LiveKit example project, and the Flutter WebRTC example project. I tested these projects on my iPhone XR, using the H264 encoder at a 540_43 resolution and 20 fps.

I also experimented with the example project for the camera package. It appears to utilize approximately 30% CPU when the camera is active. Is it typical for the camera in Flutter for iOS to demand such high CPU consumption?

Also, a big thanks for everything you do. @cloudwebrtc

[EqualCHZH]

I have the same heating problem. iPhone users complain about the application overheating and overheating after 30 minutes of use.

@cloudwebrtc @davidzhao

[domvie]

Can confirm, same thing happens to our iOS devices (12, 14 Pro, iPad Pro 11"). Very high energy + CPU usage. Perhaps this is a flutter-webrtc issue?

[elonmaph]

high cpu ,high energy and high temperature, Is there any official plan to solve this performance problem?

[cloudwebrtc]

Hey all, we will investigate this issue this week.

[cloudwebrtc]

Can confirm, same thing happens to our iOS devices (12, 14 Pro, iPad Pro 11"). Very high energy + CPU usage. Perhaps this is a flutter-webrtc issue?

hi @domvie Could you provide some reference for cpu usage? For example, open the camera on 14 pro, publish a 720p video track with using h264, I have a 14 pro, and iPad air M1, I will do a comparison test

[domvie]

Can confirm, same thing happens to our iOS devices (12, 14 Pro, iPad Pro 11"). Very high energy + CPU usage. Perhaps this is a flutter-webrtc issue?

hi @domvie Could you provide some reference for cpu usage? For example, open the camera on 14 pro, publish a 720p video track with using h264, I have a 14 pro, and iPad air M1, I will do a comparison test

So I just re-tested this using the livekit client-sdk-flutter example (I believe its version 2.1.0 as of right now). I connected with two real clients (iPhone 12, iPad Pro 11"). At times I've played around with adding a couple of video publishers using livekit-cli but that never seemed to make a difference.

Some stats I'm getting:

• H264/720p/30fps/Simulcast/Adaptive Stream/Dynacast after ~10min: iPhone 12: CPU ~50%, Energy Impact: "Very High" (with GPU taking around ~45% and "Overhead" the next largest 25%). No overheating. iPad: Virtually the same stats, also no overheating

• Same settings as above but with E2EE: Again pretty much same as above, except CPU spikes to around ~60%, no overheating.

• Same settings as above but 1080p and simulcast off: No difference, no overheating.

• Everything off, 1080p/30fps (and I assume still h264 by default?): Again all the same and no overheating.

• VP8, 1080p/30fps, everything enabled again: iPhone 12: CPU goes up to around 130% iPad: CPU ~65% quite expected. The devices, especially iPhone, do get noticeably warmer this time. Not so much that I get a OS warning about it though (which honestly was the biggest issue, even worse than the extreme throttling that would happen sometimes), but it's very warm.

• AV1, 1080p/30fps, again everything enabled: CPU goes up substantially, to around 200-300% especially when moving camera. Still warm/hot, some throttling, but not yet overheating.

At this point I have stopped testing as it is quite time consuming.

All in all, in this session I could unfortunately not reproduce the overheating issue I encountered a while back. Especially when using hardware accelerated H264 the devices seemed to run just fine, other codecs increased CPU and heat quite a bit (although that is to be expected). Energy consumption was always very high.

I'm guessing this could be due to multiple reasons:

• not testing long enough with various settings/combinations
• perhaps the problem lies somewhere in the implementation of livekit in our app, or some UI/Flutter component triggering it
• different version? (was tested on v2.0 before)

In the current form of our app it also does not overheat most of the time. We enable only dynacast, everything else off, with a resolution of 1440p @ 30fps and max bitrate of 15mbps using H264. These are the settings we had the most success with for our requirements. So it can definitely handle quite a bit of data without causing issues. I also remember the problem happening independent of res/fps, so it would also overheat at say 540p and a low frame rate. The weirdest thing to me is that it's not easily reproducible, or why it would sometimes happen with h264 and sometimes not. Other codecs make more sense I guess but even then it was never obvious.

Hopefully this was somehow helpful, sorry I couldn't reproduce it this time. Perhaps @furkanKotic still can.

[cloudwebrtc]

findings

1. Flutter’s own texture rendering will cause basic CPU usage, take iPhone 14 pro max as an example When the camera is turned on but video rendering is not performed, the CPU usage is about 3~4%.

Turn on the camera but return a nil pointer from copyPixelBuffer, which will trigger flutter UI Widgets to refresh. At this time, the CPU usage will reach 18~22%

Normally rendering the camera stream, due to YUV rotation and conversion to RGBA format texture data, the CPU usage will reach 28~34%

2. The same VideoTrack uses PlatformView + RTCMTLVideoView to render and the CPU usage is 10%.

loopback test 1280x720 30fps Using Flutter Texture to render CPU usage is about 47~55%

Using Flutter PlatformView + RTCMTLVideoView, the rendering CPU usage is about 30～35%

Related PR drafts https://github.com/flutter-webrtc/flutter-webrtc/pull/1569

[elonmaph]

should I just upgrade the sdk to 2.1.5 to solve this issue?

[cloudwebrtc]

@elonmaph you can try this branch https://github.com/livekit/client-sdk-flutter/pull/509, and change VideoTrackRenderer like this.

VideoTrackRenderer(
    _videoTrack!,
    fit: RTCVideoViewObjectFit.RTCVideoViewObjectFitContain,
    renderMode: VideoRenderMode.platformView,
  )

[dragontran]

I've been testing the platform view branch and wanted to share some feedback. The CPU performance improvements are definitely noticeable and my iphone hasn’t overheated, which is fantastic!

However, I’ve encountered a couple of issues:

• The platform views do not respect the RTCVideoViewObjectFit.RTCVideoViewObjectFitCover fit option and extend beyond the widget frame.
• The video feed briefly freezes when muting/unmuting the microphone. This issue worsens when rapidly toggling mute/unmute.

I’m really excited about the platform view and performance updates for iOS! Thanks so much for your help!

[furkanKotic]

• The platform views do not respect the RTCVideoViewObjectFit.RTCVideoViewObjectFitCover fit option and extend beyond the widget frame.

I noticed this problem too. It's like it doesn't care about the height of the Sizedbox and the image is created with a size higher than the value given to the Sizedbox.