jitsi-videobridge
jitsi-videobridge copied to clipboard
jitsi
Increasing memory comsumption in Videobridge
In a production run of Videobridge, we observe an increasing consumption of memory. Without any activity, the instance that started with 250 Mb of heap ends with about 500 Mb at the end of the day.
After obtaining a heap dump with jmap and examining it with visualvm, we find that memory is retained in the buffer pool (PartitionedByteBufferPool) in large amounts, memory is increased and never released.
The reason appears to be a small mistake in the size of the byte arrays. Visualvm shows that the byte arrays stored are frequently of size 1244, 1243 and 1242, but the maximum expected byte array in src/main/java/org/jitsi/videobridge/util/ByteBufferPool.java is 1240.
private static int T1 = 220;
private static int T2 = 775;
private static int T3 = 1240;
As a quick fix, please replace 1240 by 1244.
Thanks for this report, we recently saw we're encountering the same issue. A fix will likely be merged today (slightly different than what you have in #1484 , but thank you for the submission)
I can tell you that our problem in production was gone with the threshold of 1244.
I would have place a threshold of 1244 and perhaps an additional threshold of 2000, just in case some large buffer appears. Otherwise there is some waste of memory.
Just out of curiosity, we observe this issue only after the last upgrade. When was it introduced?
We didn't make any change in our system - our current hypothesis is that Chrome started sending bigger packets with the release of Chrome 86.
We decided to make the threshold bigger than the 1244 minimum so that we'd be robust against future further increases in the packet sizes that browsers send.
during our test, I can see ByteBufferPool upto 3000+... is this a memory leak? meaning jvb won't be able to recover the heap memory during garbage collection? @bbaldino
during our test, I can see ByteBufferPool upto 3000+... is this a memory leak? meaning jvb won't be able to recover the heap memory during garbage collection? @bbaldino
If the pool is growing, then it's likely not a leak as it means the memory is being returned to the pool. Note that the pool doesn't shrink, so whatever it grows to it will remain there. @bgrozev and @JonathanLennox were discussing some possible strategies to shrink the pool, but I don't think any changes have been made.
so it is possible that jvb heap memory (default 3GB) will be consumed by this buffer allocation when it received a continuous large buffer that is more than 1500 bytes?
do you have any strategy why the application/browser sends a large buffer? Is there something to do with config.js used by the application?
so it is possible that jvb heap memory (default 3GB) will be consumed by this buffer allocation when it received a continuous large buffer that is more than 1500 bytes?
I think it's very unlikely a single large buffer would exhaust this, as things like the MTU end up being practical limitations on the sizes we allocate there.
do you have any strategy why the application/browser sends a large buffer? Is there something to do with config.js used by the application?
Are you saying you saw a single large buffer allocated? Or many instances of regular-sized buffers?
@gtena378 You should obtain a heap dump with jmap. Then, browsing it with visualvm, one can see where is memory being consumed.
Without data, it is imposible to know what is happening with memory.
so it is possible that jvb heap memory (default 3GB) will be consumed by this buffer allocation when it received a continuous large buffer that is more than 1500 bytes?
I think it's very unlikely a single large buffer would exhaust this, as things like the MTU end up being practical limitations on the sizes we allocate there.
do you have any strategy why the application/browser sends a large buffer? Is there something to do with config.js used by the application?
Are you saying you saw a single large buffer allocated? Or many instances of regular-sized buffers?
I mean we saw a lot of like this in jvb.log:
on different sizes, some size is larger than 3000
@gtena378 You should obtain a heap dump with jmap. Then, browsing it with visualvm, one can see where is memory being consumed.
Without data, it is imposible to know what is happening with memory.
did I missed something?
now we collected heap dump but can't identify if there's a memory leak
during our testing I've seen a lots of warning log of bytebufferpool
at that moment we tried to get the live heap dump around 3.6GB but it was corrupted due to lack of disk space after fixing the disk space (after several hours) we captured another live heap dump and it was only 2.4GB
We encountered out of memory here's the details: https://community.jitsi.org/t/how-to-force-jvb2-take-momory-more-than-4gb/28143/7
here's the heap dump:
there's a lot of byte[] and ArrayCache$Container instance
We tried to change the T3 size in ByteBufferPool from 1500 to 2700 and compiled it. During our crowd testing with 28 participants we didn't encountered ByteBufferPool warning but still we encountered cpu spike and out of memory. Here's the heap dump:
