srs
srs copied to clipboard
Published
20 hours ago •
ossrs
ossrs
Memory leak observation SRS 4.0
注意：不提供以下信息的Issue会被直接删除(Please follow issue template, or we will delete it)
注意：咨询和讨论请提交到SRS星球(Please ask question at) http://bbs.ossrs.net
描述(Description)
描述你遇到了什么问题(Please description your issue here)
Memory leak in SRS when running as a an RTMP service inside a pod on k8s (maybe the memory leak is present when running SRS outside of k8s too?)
We are monitoring CPU and memory usage of SRS via the http_api which is very useful, however we notice that while cpu does return to pre-stream level, memory remains high, see attached graphs for a single RTMP stream running on SRS:
I attached memleax to the process and can see memory which could be freed (maybe this helps?):
CallStack[232]: memory expires with 48 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00000000005097b3 srs _ZN9__gnu_cxx13new_allocatorISt13_Rb_tree_nodeISt4pairIKlP19SrsSharedPtrMessageEEE8allocateEmPKv()+73 e/c++/4.8.2/new:115
0x0000000000509416 srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE11_M_get_nodeEv()+34 e/c++/4.8.2/new:115
0x0000000000508aef srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE14_M_create_nodeIJS0_IlS3_EEEEPSt13_Rb_tree_nodeIS4_EDpOT_()+29 e/c++/4.8.2/new:115
0x000000000050789b srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE10_M_insert_IS0_IlS3_EEESt17_Rb_tree_iteratorIS4_EPSt18_Rb_tree_node_baseSG_OT_()+169 e/c++/4.8.2/new:115
0x00000000005063b6 srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE15_M_insert_equalIS0_IlS3_EEESt17_Rb_tree_iteratorIS4_EOT_()+96 e/c++/4.8.2/new:115
0x0000000000505742 srs _ZNSt8multimapIlP19SrsSharedPtrMessageSt4lessIlESaISt4pairIKlS1_EEE6insertIS4_IlS1_EvEESt17_Rb_tree_iteratorIS6_EOT_()+46 e/c++/4.8.2/new:115
0x00000000004fb7b2 srs _ZN11SrsMixQueue4pushEP19SrsSharedPtrMessage()+70 e/c++/4.8.2/new:115
0x0000000000502aa4 srs _ZN13SrsLiveSource8on_audioEP16SrsCommonMessage()+426 e/c++/4.8.2/new:115
0x00000000004f5f8d srs _ZN11SrsRtmpConn23process_publish_messageEP13SrsLiveSourceP16SrsCommonMessage()+203 e/c++/4.8.2/new:115
0x00000000004f5e3b srs _ZN11SrsRtmpConn22handle_publish_messageEP13SrsLiveSourceP16SrsCommonMessage()+825 e/c++/4.8.2/new:115
0x00000000005a5e1e srs _ZN20SrsPublishRecvThread7consumeEP16SrsCommonMessage()+254 e/c++/4.8.2/new:115
0x00000000005a507c srs _ZN13SrsRecvThread8do_cycleEv()+266 e/c++/4.8.2/new:115
0x00000000005a4ee8 srs _ZN13SrsRecvThread5cycleEv()+112 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x00000000030bcc60 ??
CallStack[227]: memory expires with 40 bytes, 6 times again
CallStack[227]: memory expires with 40 bytes, 7 times again
CallStack[224]: memory expires with 901 bytes, 17 times again
CallStack[226]: memory expires with 48 bytes, 10 times again
CallStack[231]: memory expires with 40 bytes, 2 times again
CallStack[232]: memory expires with 48 bytes, 2 times again
CallStack[228]: memory expires with 40 bytes, 8 times again
CallStack[228]: memory expires with 40 bytes, 9 times again
CallStack[224]: memory expires with 5 bytes, 18 times again
CallStack[225]: memory expires with 48 bytes, 8 times again
CallStack[233]: memory expires with 40 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x000000000041e79f srs _ZN19SrsSharedPtrMessage5copy2Ev()+23 e/c++/4.8.2/new:115
0x000000000041e75f srs _ZN19SrsSharedPtrMessage4copyEv()+73 e/c++/4.8.2/new:115
0x000000000050342a srs _ZN13SrsLiveSource8on_videoEP16SrsCommonMessage()+686 e/c++/4.8.2/new:115
0x00000000004f6008 srs _ZN11SrsRtmpConn23process_publish_messageEP13SrsLiveSourceP16SrsCommonMessage()+326 e/c++/4.8.2/new:115
0x00000000004f5e3b srs _ZN11SrsRtmpConn22handle_publish_messageEP13SrsLiveSourceP16SrsCommonMessage()+825 e/c++/4.8.2/new:115
0x00000000005a5e1e srs _ZN20SrsPublishRecvThread7consumeEP16SrsCommonMessage()+254 e/c++/4.8.2/new:115
0x00000000005a507c srs _ZN13SrsRecvThread8do_cycleEv()+266 e/c++/4.8.2/new:115
0x00000000005a4ee8 srs _ZN13SrsRecvThread5cycleEv()+112 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x00000000030bcc60 ??
CallStack[234]: memory expires with 48 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00000000005097b3 srs _ZN9__gnu_cxx13new_allocatorISt13_Rb_tree_nodeISt4pairIKlP19SrsSharedPtrMessageEEE8allocateEmPKv()+73 e/c++/4.8.2/new:115
0x0000000000509416 srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE11_M_get_nodeEv()+34 e/c++/4.8.2/new:115
0x0000000000508aef srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE14_M_create_nodeIJS0_IlS3_EEEEPSt13_Rb_tree_nodeIS4_EDpOT_()+29 e/c++/4.8.2/new:115
0x000000000050789b srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE10_M_insert_IS0_IlS3_EEESt17_Rb_tree_iteratorIS4_EPSt18_Rb_tree_node_baseSG_OT_()+169 e/c++/4.8.2/new:115
0x00000000005063b6 srs _ZNSt8_Rb_treeIlSt4pairIKlP19SrsSharedPtrMessageESt10_Select1stIS4_ESt4lessIlESaIS4_EE15_M_insert_equalIS0_IlS3_EEESt17_Rb_tree_iteratorIS4_EOT_()+96 e/c++/4.8.2/new:115
0x0000000000505742 srs _ZNSt8multimapIlP19SrsSharedPtrMessageSt4lessIlESaISt4pairIKlS1_EEE6insertIS4_IlS1_EvEESt17_Rb_tree_iteratorIS6_EOT_()+46 e/c++/4.8.2/new:115
0x00000000004fb7b2 srs _ZN11SrsMixQueue4pushEP19SrsSharedPtrMessage()+70 e/c++/4.8.2/new:115
0x0000000000503440 srs _ZN13SrsLiveSource8on_videoEP16SrsCommonMessage()+708 e/c++/4.8.2/new:115
0x00000000004f6008 srs _ZN11SrsRtmpConn23process_publish_messageEP13SrsLiveSourceP16SrsCommonMessage()+326 e/c++/4.8.2/new:115
0x00000000004f5e3b srs _ZN11SrsRtmpConn22handle_publish_messageEP13SrsLiveSourceP16SrsCommonMessage()+825 e/c++/4.8.2/new:115
0x00000000005a5e1e srs _ZN20SrsPublishRecvThread7consumeEP16SrsCommonMessage()+254 e/c++/4.8.2/new:115
0x00000000005a507c srs _ZN13SrsRecvThread8do_cycleEv()+266 e/c++/4.8.2/new:115
0x00000000005a4ee8 srs _ZN13SrsRecvThread5cycleEv()+112 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x00000000030bcc60 ??
CallStack[229]: memory expires with 40 bytes, 4 times again
CallStack[229]: memory expires with 40 bytes, 5 times again
CallStack[235]: memory expires with 34 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x000000000041662f srs _ZN12SrsCplxError6createEPKcS1_iiS1_z()+243
0x00000000005232fe srs _ZN16SrsFastCoroutine9interruptEv()+110 e/c++/4.8.2/new:115
0x00000000005231c1 srs _ZN16SrsFastCoroutine4stopEv()+49 e/c++/4.8.2/new:115
0x0000000000522d38 srs _ZN14SrsSTCoroutine4stopEv()+28 e/c++/4.8.2/new:115
0x00000000005cb2c7 srs _ZN9SrsNgExec12on_unpublishEv()+43
0x00000000004fdc3e srs _ZN12SrsOriginHub12on_unpublishEv()+202 e/c++/4.8.2/new:115
0x0000000000504406 srs _ZN13SrsLiveSource12on_unpublishEv()+72 e/c++/4.8.2/new:115
0x00000000004f5aff srs _ZN11SrsRtmpConn15release_publishEP13SrsLiveSource()+87 e/c++/4.8.2/new:115
0x00000000004f4b2b srs _ZN11SrsRtmpConn10publishingEP13SrsLiveSource()+803 e/c++/4.8.2/new:115
0x00000000004f235c srs _ZN11SrsRtmpConn20stream_service_cycleEv()+3750 e/c++/4.8.2/new:115
0x00000000004f11d0 srs _ZN11SrsRtmpConn13service_cycleEv()+1474 e/c++/4.8.2/new:115
0x00000000004efe1d srs _ZN11SrsRtmpConn8do_cycleEv()+2151 e/c++/4.8.2/new:115
0x00000000004f8012 srs _ZN11SrsRtmpConn5cycleEv()+50 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002fcb350 ??
CallStack[236]: memory expires with 47 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x000000000041664c srs _ZN12SrsCplxError6createEPKcS1_iiS1_z()+272
0x00000000005232fe srs _ZN16SrsFastCoroutine9interruptEv()+110 e/c++/4.8.2/new:115
0x00000000005231c1 srs _ZN16SrsFastCoroutine4stopEv()+49 e/c++/4.8.2/new:115
0x0000000000522d38 srs _ZN14SrsSTCoroutine4stopEv()+28 e/c++/4.8.2/new:115
0x00000000005cb2c7 srs _ZN9SrsNgExec12on_unpublishEv()+43
0x00000000004fdc3e srs _ZN12SrsOriginHub12on_unpublishEv()+202 e/c++/4.8.2/new:115
0x0000000000504406 srs _ZN13SrsLiveSource12on_unpublishEv()+72 e/c++/4.8.2/new:115
0x00000000004f5aff srs _ZN11SrsRtmpConn15release_publishEP13SrsLiveSource()+87 e/c++/4.8.2/new:115
0x00000000004f4b2b srs _ZN11SrsRtmpConn10publishingEP13SrsLiveSource()+803 e/c++/4.8.2/new:115
0x00000000004f235c srs _ZN11SrsRtmpConn20stream_service_cycleEv()+3750 e/c++/4.8.2/new:115
0x00000000004f11d0 srs _ZN11SrsRtmpConn13service_cycleEv()+1474 e/c++/4.8.2/new:115
0x00000000004efe1d srs _ZN11SrsRtmpConn8do_cycleEv()+2151 e/c++/4.8.2/new:115
0x00000000004f8012 srs _ZN11SrsRtmpConn5cycleEv()+50 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002fcb350 ??
CallStack[237]: memory expires with 36 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x0000000000416694 srs _ZN12SrsCplxError6createEPKcS1_iiS1_z()+344
0x00000000005232fe srs _ZN16SrsFastCoroutine9interruptEv()+110 e/c++/4.8.2/new:115
0x00000000005231c1 srs _ZN16SrsFastCoroutine4stopEv()+49 e/c++/4.8.2/new:115
0x0000000000522d38 srs _ZN14SrsSTCoroutine4stopEv()+28 e/c++/4.8.2/new:115
0x00000000005cb2c7 srs _ZN9SrsNgExec12on_unpublishEv()+43
0x00000000004fdc3e srs _ZN12SrsOriginHub12on_unpublishEv()+202 e/c++/4.8.2/new:115
0x0000000000504406 srs _ZN13SrsLiveSource12on_unpublishEv()+72 e/c++/4.8.2/new:115
0x00000000004f5aff srs _ZN11SrsRtmpConn15release_publishEP13SrsLiveSource()+87 e/c++/4.8.2/new:115
0x00000000004f4b2b srs _ZN11SrsRtmpConn10publishingEP13SrsLiveSource()+803 e/c++/4.8.2/new:115
0x00000000004f235c srs _ZN11SrsRtmpConn20stream_service_cycleEv()+3750 e/c++/4.8.2/new:115
0x00000000004f11d0 srs _ZN11SrsRtmpConn13service_cycleEv()+1474 e/c++/4.8.2/new:115
0x00000000004efe1d srs _ZN11SrsRtmpConn8do_cycleEv()+2151 e/c++/4.8.2/new:115
0x00000000004f8012 srs _ZN11SrsRtmpConn5cycleEv()+50 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002fcb350 ??
CallStack[238]: memory expires with 96 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x0000000000416928 srs _ZN12SrsCplxError4copyEPS_()+42
0x00000000005234fe srs _ZN16SrsFastCoroutine4pullEv()+48 e/c++/4.8.2/new:115
0x0000000000522d74 srs _ZN14SrsSTCoroutine4pullEv()+28 e/c++/4.8.2/new:115
0x00000000005cb316 srs _ZN9SrsNgExec5cycleEv()+54
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[239]: memory expires with 96 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00000000004167d3 srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+197
0x00000000005cb34c srs _ZN9SrsNgExec5cycleEv()+108
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[240]: memory expires with 30 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x0000000000416802 srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+244
0x00000000005cb34c srs _ZN9SrsNgExec5cycleEv()+108
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[241]: memory expires with 52 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x000000000041681f srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+273
0x00000000005cb34c srs _ZN9SrsNgExec5cycleEv()+108
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[242]: memory expires with 38 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x0000000000416875 srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+359
0x00000000005cb34c srs _ZN9SrsNgExec5cycleEv()+108
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[243]: memory expires with 96 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00000000004167d3 srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+197
0x000000000052341a srs _ZN16SrsFastCoroutine5cycleEv()+240 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[244]: memory expires with 30 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x0000000000416802 srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+244
0x000000000052341a srs _ZN16SrsFastCoroutine5cycleEv()+240 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[245]: memory expires with 47 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x000000000041681f srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+273
0x000000000052341a srs _ZN16SrsFastCoroutine5cycleEv()+240 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[246]: memory expires with 40 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x00007fcb80f39cd9 libstdc++.so _ZNSs4_Rep9_S_createEmmRKSaIcE()+89
0x00007fcb80f39e96 libstdc++.so _ZNSs9_M_mutateEmmm()+102
0x00007fcb80f3a45e libstdc++.so _ZNSs15_M_replace_safeEmmPKcm()+30
0x0000000000416875 srs _ZN12SrsCplxError4wrapEPKcS1_iPS_S1_z()+359
0x000000000052341a srs _ZN16SrsFastCoroutine5cycleEv()+240 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002f7de90 ??
CallStack[247]: memory expires with 40 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x000000000041e79f srs _ZN19SrsSharedPtrMessage5copy2Ev()+23 e/c++/4.8.2/new:115
0x000000000041e75f srs _ZN19SrsSharedPtrMessage4copyEv()+73 e/c++/4.8.2/new:115
0x00000000004ff831 srs _ZN12SrsMetaCache19update_previous_vshEv()+125 e/c++/4.8.2/new:115
0x0000000000504450 srs _ZN13SrsLiveSource12on_unpublishEv()+146 e/c++/4.8.2/new:115
0x00000000004f5aff srs _ZN11SrsRtmpConn15release_publishEP13SrsLiveSource()+87 e/c++/4.8.2/new:115
0x00000000004f4b2b srs _ZN11SrsRtmpConn10publishingEP13SrsLiveSource()+803 e/c++/4.8.2/new:115
0x00000000004f235c srs _ZN11SrsRtmpConn20stream_service_cycleEv()+3750 e/c++/4.8.2/new:115
0x00000000004f11d0 srs _ZN11SrsRtmpConn13service_cycleEv()+1474 e/c++/4.8.2/new:115
0x00000000004efe1d srs _ZN11SrsRtmpConn8do_cycleEv()+2151 e/c++/4.8.2/new:115
0x00000000004f8012 srs _ZN11SrsRtmpConn5cycleEv()+50 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002fcb350 ??
CallStack[248]: memory expires with 40 bytes, backtrace:
0x00007fcb8061b6b0 libc-2.17.so malloc()+0
0x00007fcb80edb18d libstdc++.so _Znwm()+29
0x000000000041e79f srs _ZN19SrsSharedPtrMessage5copy2Ev()+23 e/c++/4.8.2/new:115
0x000000000041e75f srs _ZN19SrsSharedPtrMessage4copyEv()+73 e/c++/4.8.2/new:115
0x00000000004ff8bf srs _ZN12SrsMetaCache19update_previous_ashEv()+125 e/c++/4.8.2/new:115
0x0000000000504475 srs _ZN13SrsLiveSource12on_unpublishEv()+183 e/c++/4.8.2/new:115
0x00000000004f5aff srs _ZN11SrsRtmpConn15release_publishEP13SrsLiveSource()+87 e/c++/4.8.2/new:115
0x00000000004f4b2b srs _ZN11SrsRtmpConn10publishingEP13SrsLiveSource()+803 e/c++/4.8.2/new:115
0x00000000004f235c srs _ZN11SrsRtmpConn20stream_service_cycleEv()+3750 e/c++/4.8.2/new:115
0x00000000004f11d0 srs _ZN11SrsRtmpConn13service_cycleEv()+1474 e/c++/4.8.2/new:115
0x00000000004efe1d srs _ZN11SrsRtmpConn8do_cycleEv()+2151 e/c++/4.8.2/new:115
0x00000000004f8012 srs _ZN11SrsRtmpConn5cycleEv()+50 e/c++/4.8.2/new:115
0x00000000005233e9 srs _ZN16SrsFastCoroutine5cycleEv()+191 e/c++/4.8.2/new:115
0x000000000052346c srs _ZN16SrsFastCoroutine3pfnEPv()+32 e/c++/4.8.2/new:115
0x000000000066397f srs _st_thread_main()+40 sched.c:363
0x00000000006641f2 srs st_thread_create()+306 sched.c:694
0x0000000002fcb350 ??
CallStack[224]: expired-memory frees after 31 seconds
CallStack[225]: expired-memory frees after 31 seconds
CallStack[227]: expired-memory frees after 31 seconds
CallStack[224]: expired-memory frees after 31 seconds
CallStack[226]: expired-memory frees after 31 seconds
CallStack[228]: expired-memory frees after 31 seconds
CallStack[224]: expired-memory frees after 31 seconds
Warning: too many expired-free at CallStack[224]. will not show this CallStack later
I am going to try some older container versions of 4.0 to see if the problem exists there also.
- SRS版本(Version):
4.0.167 - SRS的日志如下(Log):
[2021-10-12 09:39:43.903][Trace][17][87276367] XCORE-SRS/4.0.167(Leo)
[2021-10-12 09:39:43.903][Trace][17][87276367] config parse complete
[2021-10-12 09:39:43.903][Trace][17][87276367] write log to console
[2021-10-12 09:39:43.904][Warn][17][87276367][22] stats network use index=0, ip=10.8.13.11, ifname=enp193s0
[2021-10-12 09:39:43.904][Warn][17][87276367][22] stats disk not configed, disk iops disabled.
[2021-10-12 09:39:43.904][Warn][17][87276367][22] SRS/4.0.167 is not stable
[2021-10-12 09:39:43.908][Warn][17][yq40fgq3][22] use public address as ip: 157.XX.XX.XX, ifname=enp195s0
- SRS的配置如下(Config):
listen 157.XX.XX.XX:1935 10.8.13.3:1935;
max_connections 10000;
srs_log_tank console;
srs_log_level warn;
daemon off;
http_api {
enabled on;
listen 1985;
}
http_server {
enabled on;
listen 1995;
dir ./objs/nginx/html;
}
srt_server {
enabled on;
listen 10080;
maxbw 1000000000;
connect_timeout 4000;
peerlatency 300;
recvlatency 300;
}
vhost __defaultVhost__ {
http_hooks {
enabled on;
on_publish http://10.8.13.3:8089/srs/on_publish/;
on_unpublish http://10.8.13.3:8089/srs/on_unpublish/;
on_play http://10.8.13.3:8089/srs/on_play/;
}
publish {
firstpkt_timeout 30000;
normal_timeout 15000;
}
}
重现(Replay)
重现Bug的步骤(How to replay bug?)
- Run SRS docker image
- Observe memory usage by curl http_api
- Connect a single RTMP stream
- Observe memory usage again by curl http_api
- Disconnect single RTMP stream
- Observe memory usage again by curl http_api
期望行为(Expect)
描述你期望发生的事情(Please describe your expectation)
I would expect memory usage after a stream has ended to return to roughly the same level as prior to the stream beginning.
It maybe cache, not leak. Please run for a long time, for example, 10 days, and it leaks if the memory grow all time. It's cache if it grow and stop at some level of memory.
OK that's interesting - we have run for a few days and had to increase the memory limit on the SRS pods as they were being OOM killed, in this graph you can see even when the memory limit was increased to 4Gb, one pod still got OOM killed
What is also interesting is that the increase in memory usage appears to be stepped and in some cases quite dramatic
Is there a way of limiting the cache or clearing the cache? Forgive me if I have missed this in some docs somewhere.
We believe this memory issues occurs when our stream validation system validates a stream key - but the stream does not contain a valid video stream, so only audio is streamed - our transcoding pipeline then rejects the stream, but because the validation server already accepted the connection srs continues to stream the data, but nothing pulls from it...
@winlinvip thanks for looking at this one, an observation we have is that the 'memory leak' happens when a bad stream is present on SRS - when I say bad I mean that the stream was not accepted by our validation system because the user's stream key was invalid - at this point we try to disconnect them using the SRS /api/v1/clients endpoint - for most user streams this works - but for some (probably audio only - as the rate of growth of memory indicates around 300kbps stream bit rate) who have persistent OBS (or some other software/restream services) connection retries SRS does not always fully disconnect them, one option I thought of is that if SRS had a way to block/deny based on stream key - in a similar way to ip deny/allow list in the security section?
We are still suffering from this memory issue - it seems to be related to streams which throw the warning/error:
clear gop cache for guess pure audio overflow
Which I thought meant that the stream had no video - I tested this theory by using Wirecast and setting the encoder to only broadcast the audio portion of the RTMP stream - I did not see any of the above warning/error messages in the log even though there was no video
I'm amazed that more users have not reported this issue?
It seems some strange stream causes this issue, the gop cache can't detect the duration of audio frames.
- You can disable the gop cache by
gop_cache off;, note that without gop cache, viewer wait longer to see the first frame. - If you need gop cache, you can try to replay this issue, seems it's caused by some pure audio stream, with incorrect timestamp.
It's not a general use case for memoty leak, so it's not common.
This is a still a pretty major problem for us, is there any chance you could be persuaded to look at it. We see an SRS instance around once a week do this behaviour. When it happens the node its on slows down all the other connections, introducing buffering up to 100s of clients can be impacted causing a massive degradation of the service.
The problem then becomes worse than this, because the offending user then reconnects and takes out another SRS node, this then repeats and thousands of streams can become impacted quickly, causing a service wide outage. When you combine this with the lack of ip blocking, it leaves SRS vulnerable to a DOS type attack making it very hard to build a reliable User facing service.
This issue does not appear to be a memory leak, but rather an increase in cache size due to abnormal content, such as unrecognizable audio and video packets causing the GOP (Group of Pictures) cache to grow. Since this involves specific audio and video content, a detailed examination of the content in question is necessary.
A relatively simple method to circumvent the issue is to isolate these users. Alternatively, one could isolate the normal users instead, such as by directing key users to a separate, isolated SRS (Simple Real-time Server) node.
Because the content is related to the users, network isolation is a relatively effective method.
TRANS_BY_GPT4