bugtracker icon indicating copy to clipboard operation
bugtracker copied to clipboard
verified publisher icon Kurento

Kurento crash with error

Open nightwolftamb opened this issue 6 years ago • 10 comments

KMS Version:

6.13.1~1.g5ef465a Build from sources (changed resolution to 1280x720 and FPS to 30 in compositor)

Ubuntu Version

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

Other libraries versions:

ii gir1.2-gst-plugins-base-1.5 1.8.1-1kurento1.18.04 amd64 Description: GObject introspection data for the GStreamer Plugins Base library ii gir1.2-gstreamer-1.5 1.8.1-1kurento2.18.04~20191018180342.gbp84c84b amd64 Description: GObject introspection data for the GStreamer library ii gir1.2-nice-0.1:amd64 0.1.15-1kurento3.18.04 amd64 ICE library (GObject introspection) ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.18.04 amd64 libav plugin for GStreamer ii gstreamer1.5-nice:amd64 0.1.15-1kurento3.18.04 amd64 ICE library (GStreamer 1.5 plugin) ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento4.18.04~20191018180342.gbp17d48d amd64 GStreamer plugins from the "bad" set ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "base" set ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins from the "good" set ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "ugly" set ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugin for PulseAudio ii gstreamer1.5-x:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins for X11 and Pango ii kms-cmake-utils 6.12.1-0kurento1.18.04~20191018180342.gbp91cc3e all Kurento CMake utils ii kms-cmake-utils-build-deps 6.12.1-0kurento1 all build-dependencies for kms-cmake-utils ii kms-core 6.12.1-0kurento1.18.04~20191018180342.gbpb345e3 amd64 Kurento Core module ii kms-core-build-deps 6.12.1-0kurento1 all build-dependencies for kms-core ii kms-core-dev 6.12.1-0kurento1.18.04~20191018180342.gbpb345e3 amd64 Kurento Core module ii kms-elements 6.12.1-0kurento1.18.04~20191018180342.gbpf19f0b amd64 Kurento Elements module ii kms-elements-build-deps 6.12.1-0kurento1 all build-dependencies for kms-elements ii kms-elements-dev 6.12.1-0kurento1.18.04~20191018180342.gbpf19f0b amd64 Kurento Elements module ii kms-filters 6.12.1-0kurento1.18.04~20191018180342.gbpbbef4a amd64 Kurento Filters module ii kms-filters-build-deps 6.12.1-0kurento1 all build-dependencies for kms-filters ii kms-filters-dev 6.12.1-0kurento1.18.04~20191018180342.gbpbbef4a amd64 Kurento Filters module ii kms-jsonrpc 6.12.1-0kurento1.18.04~20191018180342.gbp5017f5 amd64 Kurento JSON-RPC library ii kms-jsonrpc-build-deps 6.12.1-0kurento1 all build-dependencies for kms-jsonrpc ii kms-jsonrpc-dev 6.12.1-0kurento1.18.04~20191018180342.gbp5017f5 amd64 Kurento JSON-RPC library ii kmsjsoncpp 1.6.3-1kurento1.18.04 amd64 Kurento jsoncpp library ii kmsjsoncpp-dev 1.6.3-1kurento1.18.04 amd64 Kurento jsonrpc library ii kurento-media-server-build-deps 6.12.1-0kurento1 all build-dependencies for kurento-media-server ii kurento-module-creator 6.12.1-0kurento1.18.04~20191018180342.gbpf3922e all Kurento Module Creator ii kurento-module-creator-build-deps 6.12.1-0kurento1 all build-dependencies for kurento-module-creator ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento4.18.04~20191018180342.gbp17d48d amd64 GStreamer development files for libraries from the "bad" set ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer libraries from the "base" set ii libgstreamer-plugins-base1.5-dev 1.8.1-1kurento1.18.04 amd64 GStreamer development files for libraries from the "base" set ii libgstreamer1.5-0:amd64 1.8.1-1kurento2.18.04~20191018180342.gbp84c84b amd64 Core GStreamer libraries and elements ii libgstreamer1.5-dev 1.8.1-1kurento2.18.04~20191018180342.gbp84c84b amd64 GStreamer core development files ii libnice-dev 0.1.15-1kurento3.18.04 amd64 ICE library (development files) ii libnice10:amd64 0.1.15-1kurento3.18.04 amd64 ICE library (shared library) ii libsrtp0:amd64 1.6.0-0kurento1.18.04~20191018180342.gbp577e0a amd64 Secure RTP (SRTP) and UST Reference Implementations - shared library ii libusrsctp 0.9.2-1kurento1.18.04 amd64 sctp-refimpl library ii openh264 1.4.0-1kurento1.18.04 amd64 OpenH264 library ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento4.18.04~20191018180342.gbp17d48d amd64 GStreamer plugins from openh264 ii openwebrtc-gst-plugins 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins ii openwebrtc-gst-plugins-dev 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins

Client libraries

  • Language: Java
  • Version: openjdk version "11.0.5" 2019-10-15

Browsers tested

Not sure... It crashes one time per 2-3 weeks and there is lot of users. Will try to get more info later.

System description:

KMS located on phisical server. There is coturn server on same machine.

What steps will reproduce the problem?

Kurento just crashed with error: 2020-01-10T14:01:07,445121 88021 0x00007f7dc1ffb700 error agnosticbin kmsagnosticbin.c:397 proxy_src_pad_query_function() <src_0:proxypad6470> Caps not accepted: accept-caps query: 0x7f805c0b6720, GstQueryAcceptCaps, caps=(GstCaps)"audio/x-mulaw,\ rate=(int)48000,\ channels=(int)2", result=(boolean)false;

Here is whole log file: https://www.dropbox.com/s/vpxnwey28mnev04/2020-01-10T000038.00002.pid88021.log?dl=0

We have webinar application. User can create his own room and invite participants to it. All participants combined to one stream with Composite and send to RTMP service throw RTPEndpoint. For all endpoints we add this settings: rtp_output.setMinVideoRecvBandwidth(UserSession.MIN_VIDEO_RECV_BAND); rtp_output.setMinVideoSendBandwidth(UserSession.MIN_VIDEO_SEND_BAND); rtp_output.setMaxVideoRecvBandwidth(UserSession.MAX_VIDEO_RECV_BAND); rtp_output.setMaxVideoSendBandwidth(UserSession.MAX_VIDEO_SEND_BAND); rtp_output.setMaxAudioRecvBandwidth(UserSession.MAX_AUDIO_RECV_BAND); rtp_output.setAudioFormat(new AudioCaps(UserSession.AUDIO_CODEC, UserSession.MAX_AUDIO_RECV_BAND10241024));

public static int MIN_VIDEO_RECV_BAND = 500; public static int MIN_VIDEO_SEND_BAND = 500; public static int MAX_VIDEO_RECV_BAND = 4000; public static int MAX_VIDEO_SEND_BAND = 4000; public static int MAX_AUDIO_RECV_BAND = 500; public static AudioCodec AUDIO_CODEC = AudioCodec.PCMU;

We are using PCMU rather then OPUS because there is better quality while combining videos.

What is the expected result? Kurento not crashes

What happens instead? Kurento crashes

Does it happen with one of the tutorials? Not tested

Please provide any additional information below

nightwolftamb avatar Jan 10 '20 16:01 nightwolftamb

Some more info (maybe will help). Sometimes Kurento continue works after this error, but some processes continue use CPU. Here is error: 2020-01-11T14:35:02,102339 42166 0x00007fa929e7a700 info agnosticbin kmsagnosticbin.c:774 kms_agnostic_bin2_link_pad() Upstream provided caps: audio/x-mulaw, rate=(int)[ 8000, 192000 ], channels=(int)[ 1, 2 ] 2020-01-11T14:35:02,102416 42166 0x00007fa929e7a700 info agnosticbin kmsagnosticbin.c:783 kms_agnostic_bin2_link_pad() Downstream wanted caps: audio/x-mulaw, channels=(int)1, rate=(int)8000 2020-01-11T14:35:02,102433 42166 0x00007fa7bcff9700 error agnosticbin kmsagnosticbin.c:397 proxy_src_pad_query_function() <src_0:proxypad5140> Caps not accepted: accept-caps query: 0x7fa798003990, GstQueryAcceptCaps, caps=(GstCaps)"audio/x-mulaw,\ rate=(int)48000,\ channels=(int)2", result=(boolean)false; 2020-01-11T14:35:02,104112 42166 0x00007fa7bffff700 fixme agnosticbin kmsagnosticbin.c:1359 check_ret_error() kmsagnosticbin2-221:sink Handling flow error, fakesink message: chain ******* (agnosticbin_fakesink505:sink) (1920 bytes, dts: none, pts: 0:01:00.430000000, duration: 0:00:00.010000000, offset: 2900640, offset_end: 2901120, flags: 00000800 gap ) 0x7fa7d0673b60 Full log here: https://www.dropbox.com/s/5y0hyhef7gu62fy/2020-01-11T000015.00001.pid42166.log?dl=0

Here is the "top -H" after this issue. https://www.dropbox.com/s/7jplpiwl4n868hs/screenshot.jpg?dl=0

Will continue to check this...

nightwolftamb avatar Jan 14 '20 06:01 nightwolftamb

Will try check if this issue happens with OPUS audio codec....

nightwolftamb avatar Jan 14 '20 06:01 nightwolftamb

Not depends from codec. Same issue with OPUS.

nightwolftamb avatar Jan 19 '20 19:01 nightwolftamb

Hi, please:

  • Install debug symbols: https://doc-kurento.readthedocs.io/en/latest/dev/dev_guide.html#dev-dbg
  • If KMS crashes, have a look at errors.log
  • Try to get the crash under debugger, if you are running from sources it's easy: https://doc-kurento.readthedocs.io/en/latest/dev/dev_guide.html#building-from-sources just change the run command to this: ./bin/kms-build-run.sh --gdb. Then, you write command run to start actually executing KMS inside GDB, and if a crash happens, run bt to get a gdb backtrace, which tells us exactly where is the crash. More info here: https://doc-kurento.readthedocs.io/en/latest/user/troubleshooting.html#gstreamer-critical-messages-in-the-log

j1elo avatar Jan 19 '20 23:01 j1elo

Can't install debug symbols. Get this error:

Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: kurento-dbg : Depends: libglib2.0-0-dbg but it is not installable Depends: kmsjsoncpp-dbg but it is not going to be installed Depends: libnice10-dbgsym but it is not going to be installed Depends: openwebrtc-gst-plugins-dbg but it is not going to be installed Depends: gstreamer1.5-libav-dbg but it is not going to be installed Depends: gstreamer1.5-nice-dbgsym but it is not going to be installed Depends: gstreamer1.5-plugins-ugly-dbg but it is not going to be installed E: Unable to correct problems, you have held broken packages.

Also can't locate errors.log. There is no such file in /var/logs. How can I set the location of this file?

nightwolftamb avatar Jan 20 '20 06:01 nightwolftamb

Tried on another machine. Error: The following packages have unmet dependencies: kurento-dbg : Depends: libglib2.0-0-dbg but it is not installable E: Unable to correct problems, you have held broken packages.

Where to find "libglib2.0-0-dbg" for 18.04?

nightwolftamb avatar Jan 22 '20 10:01 nightwolftamb

Turns out the package dependencies for kurento-dbg are working for Ubuntu 16.04 but not for 18.04; I've fixed that thanks to you noticing the issue.

Updated instructions here: https://doc-kurento.readthedocs.io/en/latest/dev/dev_guide.html#dev-dbg

Note that until the fixes are published (in next Kurento release), the correct package name is libglib2.0-0-dbgsym.

This is the full list of packages to install:

libc6-dbg
libglib2.0-0-dbgsym
kmsjsoncpp-dbg
libnice10-dbgsym
openwebrtc-gst-plugins-dbg
libgstreamer1.5-0-dbg
gstreamer1.5-libav-dbg
gstreamer1.5-nice-dbgsym
gstreamer1.5-plugins-bad-dbg
gstreamer1.5-plugins-base-dbg
gstreamer1.5-plugins-good-dbg
gstreamer1.5-plugins-ugly-dbg
kms-jsonrpc-dbg
kms-core-dbg
kms-elements-dbg
kms-filters-dbg
kurento-media-server-dbg

j1elo avatar Jan 23 '20 21:01 j1elo

Now the debug tools are installed but in this mode Kurento not listening port... I am run command "./bin/kms-build-run.sh --gdb" and after that "netstat - nltp | grep kurento" OR "netstat - nltp | grep 8888" - and the result is nothing. As the result - java application can't connect to server. But the process is running - i see it with command "ps -ef | grep kurento".

When I run command "./bin/kms-build-run.sh --release" - kurento starts listening on 8888 and 8433. Not sure what I am doing wrong...

nightwolftamb avatar Jan 27 '20 06:01 nightwolftamb

Oh I made a mistake in my original instructions, sorry for that. I've now fixed it! After starting with GDB, the gdb program starts in a wait state, until you introduce some commands. So, you need to write the command run to actually start running the process. All this is explained in the linked page about running GDB: https://doc-kurento.readthedocs.io/en/latest/user/troubleshooting.html#gstreamer-critical-messages-in-the-log

Please follow instructions there to get a crash backtrace from Kurento. That will be helpful!

j1elo avatar Jan 27 '20 14:01 j1elo

I find the reason. I am streaming from webrtcendpoint to RTPEndpoint and after that stream goes to FFMPEG. Problem appeared only with this SDP offer: v=0 o=- 0 0 IN IP4 127.0.0.1 s=- c=IN IP4 127.0.0.1 t=0 0 m=audio " + current_audio_port + " RTP/AVP 96 b=AS:192 a=rtpmap:96 PCMU/8000 a=sendrecv a=ssrc:445566 cname:[email protected] m=video " + current_video_port + " RTP/AVP 103 a=rtpmap:103 H264/90000 a=sendrecv a=ssrc:112233 cname:[email protected]

Sometimes this cause kurento crash. But I am changed audio codec to opus/48000/2 and problem not happens again. Maybe this will help...

nightwolftamb avatar May 06 '20 16:05 nightwolftamb