JVB Memory Leak with high bitrate audio

[samopus1io]

There seem to be a memory leak in JVB that seems related to using higher bitrate audio (opus codec).

Jitsi latest stable_4857 + opusMaxAverageBitrate whitelist config fix https://github.com/jitsi/jitsi-meet/issues/7384

Platform used for testing:

AWS / ECS Fargate 1 task with web/jicofo/prosody (no memory or CPU issue there) (CPU 2048 / 4Gb RAM) 1 task with JVB (default memory config - 3GB limit) (CPU 2048 / 4Gb RAM)

Run with the following params: https://server_/test#config.opusMaxAverageBitrate=510000&config.stereo=true&config.disableAP=true

1 Room / 10 Participants (mostly one participant streaming audio from virtual audio interface, cameras on for all, using chrome / VP8 video)

Memory keeps increasing steadily, within 30-50 mins (depending on the number of participant, the more the quicker) until it reaches a peak at the 3GB JVM limit (~79% on the screenshots below). CPU is stable at around ~55% the whole time and Audio/video for all participants is perfect until memory limit is reached. CPU starts increasing to 80% when memory limit is reached and Audio/Video Quality starts to suffer and gets worse within a few minutes to the point that it is unusable. Memory usage doesn't go down even when participants drop (at least for about 30 mins).

AV Quality gets better when some participants drop (down to 3 or 4), but audio still drops every few seconds. Re adding a couple participants (up to 6, not even 10) and AV gets very bad again (0 frame rate on most video streams, audio on and off every couple of seconds for several seconds etc.)

Update:

Ran a few more tests with opusMaxAverageBitrate at lower bitrates (64 Kbs, 128kbs, 192kbs) and memory usage was stable not in creasing like it was with 510000, or not in a noticeable way.

Expected Behavior

---

JVB Memory usage should not keep increasing and capping out. Quality of AV should not suffer after a certain time.

[JonathanLennox]

Is there anything notable in the logs?

Out of curiosity, is there a reason you're setting the Opus bitrate so high? My expectation is that this bitrate is far above any rate necessary for acoustic transparency.

[bgrozev]

Seems like you have an environment where you can reproduce. Can you gather some additional stats as described below?

1. Restart the bridge
2. Enable gathering stats from the memory pool: curl -X POST http://localhost:8080/debug/enable/pool-stats
3. Run the test that causes the leak
4. Gather stats with:

curl http://localhost:8080/debug/stats/pool-stats
curl http://localhost:8080/debug/stats/node-stats

[samopus1io]

Thanks for looking into this.

@JonathanLennox we didn't see anything notable in the logs although they were not in debug. Also to answer your questions lower bitrate might be acceptable but for our particular use case we wanted the highest possible quality (assuming webrtc/opus would throttle and lower bitrate if bandwidth was constrained).

@bgrozev we will try to reproduce and get the stats requested. Thanks,

[elebel-landr]

is there any activity on this issue? Has anyone else seen this problem?

[amirphl]

I have still the same problem. I use jvb instances in the docker environment. I restart bridges manually at night to free memory. This is the memory usage of my jvb instances:

[MatheoJaouen]

Is it still an issue? It seems ok at audio 256 kbps but I didn't try higher since.

[amirphl]

I no longer work at that company.