webrtc
webrtc copied to clipboard
Published
20 hours ago •
webrtc-rs
webrtc-rs
Minor Memory Leak Issue for each successful peerconnection
hi
I am trying to build an app using actix-web as an ice server, ffmpeg as video/video sample feeder and this repo as a proxy.
But after I put them together I found that each peer connection will left some few memory in the system process, after a lot of heavy visits test, the process will be killed by Linux system and quit.
At first, I found some udp connection failed to get closed after peer connection closed, which can be found from bugfix-Udp connection not close (reopen #174) #195 #202
But after it was fixed and the repo version got to 0.6, I found the memory issue is still there, each time when a peer connected a portion of 0.1% of my system memory is consumed, I guess it is likely to be some threading data issue.
this is the outcome for initial state
%Cpu(s): 16.6 us, 4.1 sy, 0.0 ni, 79.2 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 7877.7 total, 117.5 free, 6234.4 used, 1525.8 buff/cache
MiB Swap: 7813.0 total, 4300.8 free, 3512.2 used. 1207.2 avail Mem
PID USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
93263 robin 20 0 2216484 25620 20060 S 19.9 0.3 0:55.53 vccplayer
And here's a few connections from web
this is what I got after 5 connection is released
%Cpu(s): 9.1 us, 5.6 sy, 0.0 ni, 85.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 7877.7 total, 178.3 free, 6186.3 used, 1513.1 buff/cache
MiB Swap: 7813.0 total, 4271.2 free, 3541.8 used. 1244.3 avail Mem
PID USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
93263 robin 20 0 2221880 64172 32396 S 26.7 0.8 2:05.10 vccplayer
no sockets left
Your see, the mem is growing, even after all udp sockets are released. Ffmpeg, however, is likely to be another reason to leak memory, so I removed ffmpeg from my code, copying the code from example play-from-disk-h264 to track the issue. Still no luck. Maybe it's because of the webrtc repo code.
To prove my point, I use this repo example play-from-disk-h264 as test subject, with an h264 and an ogg file split from a mp4 file, valgrind, --leak-check=full to test if any data is still reachable.
This is what I got
Peer Connection State has changed: closed
==77483==
==77483== HEAP SUMMARY:
==77483== in use at exit: 10,536 bytes in 73 blocks
==77483== total heap usage: 137,624 allocs, 137,551 frees, 5,498,181 bytes allocated
==77483==
==77483== 32 bytes in 1 blocks are still reachable in loss record 1 of 9
==77483== at 0x484884F: malloc (vg_replace_malloc.c:393)
==77483== by 0x12BC8CB: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x12BC956: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x12BD7E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x12BC82C: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x12B7FDE: parking_lot_core::parking_lot::HashTable::new (boxed.rs:215)
==77483== by 0x12B84D7: parking_lot_core::parking_lot::create_hashtable (parking_lot.rs:237)
==77483== by 0x12B848F: parking_lot_core::parking_lot::get_hashtable (parking_lot.rs:225)
==77483== by 0x12B85D3: parking_lot_core::parking_lot::grow_hashtable (parking_lot.rs:267)
==77483== by 0x12B8314: parking_lot_core::parking_lot::ThreadData::new (parking_lot.rs:180)
==77483== by 0x12B8DCD: parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__init (parking_lot.rs:202)
==77483== by 0x12B8E30: parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit::{{closure}} (local.rs:353)
==77483==
==77483== 128 bytes in 1 blocks are still reachable in loss record 2 of 9
==77483== at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483== by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483== by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483== by 0x1480BF9: alloc (alloc.rs:22)
==77483== by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483== by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483== by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483== by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483== by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483== by 0x6FDE73: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483==
==77483== 128 bytes in 1 blocks are still reachable in loss record 3 of 9
==77483== at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483== by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483== by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483== by 0x1480BF9: alloc (alloc.rs:22)
==77483== by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483== by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483== by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483== by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483== by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483== by 0x10DB18D: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483==
==77483== 128 bytes in 1 blocks are still reachable in loss record 4 of 9
==77483== at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483== by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483== by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483== by 0x1480BF9: alloc (alloc.rs:22)
==77483== by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483== by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483== by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483== by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483== by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483== by 0xAF36B3: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483==
==77483== 128 bytes in 1 blocks are still reachable in loss record 5 of 9
==77483== at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483== by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483== by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483== by 0x1480BF9: alloc (alloc.rs:22)
==77483== by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483== by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483== by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483== by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483== by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483== by 0xAF38ED: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483==
==77483== 128 bytes in 1 blocks are still reachable in loss record 6 of 9
==77483== at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483== by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483== by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483== by 0x1480BF9: alloc (alloc.rs:22)
==77483== by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483== by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483== by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483== by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483== by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483== by 0xAF42DD: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483==
==77483== 1,024 bytes in 1 blocks are still reachable in loss record 7 of 9
==77483== at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483== by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483== by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483== by 0x1480BF9: alloc (alloc.rs:22)
==77483== by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483== by 0x12BC8CB: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x12BC956: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x12BD7E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x12BDE20: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:185)
==77483== by 0x12BE43C: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:131)
==77483== by 0x12BA7B3: alloc::vec::Vec<T,A>::with_capacity_in (mod.rs:641)
==77483== by 0x12BA4F6: alloc::vec::Vec<T>::with_capacity (mod.rs:483)
==77483== by 0x12B7DEC: parking_lot_core::parking_lot::HashTable::new (parking_lot.rs:75)
==77483== by 0x12B84D7: parking_lot_core::parking_lot::create_hashtable (parking_lot.rs:237)
==77483==
==77483== 2,080 bytes in 1 blocks are still reachable in loss record 8 of 9
==77483== at 0x484884F: malloc (vg_replace_malloc.c:393)
==77483== by 0x12144DB: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x1214566: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x1215FE9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x11E0610: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:185)
==77483== by 0x11E4DCC: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:131)
==77483== by 0x1197E63: alloc::vec::Vec<T,A>::with_capacity_in (mod.rs:641)
==77483== by 0x1196B96: alloc::vec::Vec<T>::with_capacity (mod.rs:483)
==77483== by 0x118E5BC: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter (spec_from_iter_nested.rs:54)
==77483== by 0x119BB04: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter (spec_from_iter.rs:33)
==77483== by 0x119B5CC: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter (mod.rs:2645)
==77483== by 0x11F9BC4: core::iter::traits::iterator::Iterator::collect (iterator.rs:1792)
==77483==
==77483== 6,760 bytes in 65 blocks are still reachable in loss record 9 of 9
==77483== at 0x484884F: malloc (vg_replace_malloc.c:393)
==77483== by 0x12144DB: alloc::alloc::alloc (alloc.rs:89)
==77483== by 0x1214566: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483== by 0x1215FE9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483== by 0x121443C: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483== by 0x11BA913: alloc::sync::Arc<T>::new (boxed.rs:215)
==77483== by 0x122FA79: tokio::sync::watch::channel (watch.rs:319)
==77483== by 0x122E2CD: <tokio::signal::registry::EventInfo as core::default::Default>::default (registry.rs:22)
==77483== by 0x11F8768: <tokio::signal::unix::SignalInfo as core::default::Default>::default (unix.rs:227)
==77483== by 0x11F85F9: tokio::signal::unix::<impl tokio::signal::registry::Init for alloc::vec::Vec<tokio::signal::unix::SignalInfo>>::init::{{closure}} (unix.rs:33)
==77483== by 0x11FA010: core::iter::adapters::map::map_fold::{{closure}} (map.rs:84)
==77483== by 0x11F9B6A: core::iter::range::<impl core::iter::traits::iterator::Iterator for core::ops::range::RangeInclusive<A>>::fold::ok::{{closure}} (range.rs:1161)
==77483==
==77483== LEAK SUMMARY:
==77483== definitely lost: 0 bytes in 0 blocks
==77483== indirectly lost: 0 bytes in 0 blocks
==77483== possibly lost: 0 bytes in 0 blocks
==77483== still reachable: 10,536 bytes in 73 blocks
==77483== suppressed: 0 bytes in 0 blocks
==77483==
==77483== For lists of detected and suppressed errors, rerun with: -s
==77483== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Is this because of the tokio runtime? Or tokio task not released in your code? I'm not sure, could you examine your code to see if there's leaking issue? Thanks
I have also noticed memory leaks for each peer connection. I would suspect that there are cyclic references created by the frequent use of Arc in this project.
This project is to a large degree a 1:1 clone of the Pion server written in Go and thus shares the same software architecture. Go has a garbage collector that probably (my guess) can detect dead islands of cyclic references unlike in Rust where an island of cyclic Arc references will never be freed.
From what I understand this will be a quite a large issue for long running services that rely on WebRTC. Is there a reason the issue isn't higher priority? Nice breakdown @shiqifeng2000
