bugtracker
bugtracker copied to clipboard
Kurento
Is this normal? Two process is running and Kurento Crashed
Prerequisites
These are MANDATORY, otherwise the issue will be automatically closed.
- [x] I agree to fill this issue template.
- [x] I have read the [Troubleshooting Guide] and [Support Instructions].
Issue description
last day on a odd time, kurento stopped working, cam broadcasting is not working, until i manually restarted kurento.
Context
How to reproduce?
Here is the screenshot of htop, which shows two process running with same kurento name, with different PID.
Not sure how to reproduce the error, because the logs are kind of weird, because two log files contains nothing in it, just empty.
Attaching another file log information in here
2021-01-14T232215.00073.pid13956.log
2021-01-14T23:22:15,489620 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3888, expected 3936)
2021-01-14T23:22:15,500467 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3889, expected 3936)
2021-01-14T23:22:15,510074 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3890, expected 3936)
2021-01-14T23:22:15,519435 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3892, expected 3936)
2021-01-14T23:22:15,530267 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3893, expected 3936)
2021-01-14T23:22:15,539379 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3895, expected 3936)
2021-01-14T23:22:15,546063 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 2 ms at PTS=0:17:11.820598553 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,548457 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 2 ms at PTS=0:17:11.823001270 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,550230 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3896, expected 3936)
2021-01-14T23:22:15,554650 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 3 ms at PTS=0:17:11.829193787 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,558439 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 3 ms at PTS=0:17:11.832983584 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,560347 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3897, expected 3936)
2021-01-14T23:22:15,566080 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 3 ms at PTS=0:17:11.840627189 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,569936 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 3 ms at PTS=0:17:11.844480996 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,569972 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3899, expected 3936)
2021-01-14T23:22:15,581036 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3900, expected 3936)
2021-01-14T23:22:15,602461 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3909, expected 3937)
2021-01-14T23:22:15,613332 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3910, expected 3937)
2021-01-14T23:22:15,625298 13956 0x00007f3755941700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26751_kmsagnosticbin2-7745:sink> GAP of 66 ms at PTS=0:15:03.085070825 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,625707 13956 0x00007f3755941700 warning kmsutils kmsutils.c:478 discont_detection_probe() <kmswebrtcendpoint26751_kmsagnosticbin2-7745:sink> DISCONTINUITY at non-keyframe; will drop until keyframe
2021-01-14T23:22:15,637962 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 2 ms at PTS=0:17:11.912498535 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,656957 13956 0x00007f3777985700 warning kmsutils kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> GAP of 7 ms at PTS=0:17:11.931464803 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,657038 13956 0x00007f3777985700 warning kmsutils kmsutils.c:478 discont_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink> DISCONTINUITY at non-keyframe; will drop until keyframe
2021-01-14T23:22:15,762043 13956 0x00007f37e2a5b700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 17275, expected 17286)
2021-01-14T23:22:16,030726 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3937, expected 3985)
2021-01-14T23:22:16,069415 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3940, expected 3988)
2021-01-14T23:22:16,090610 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3944, expected 3989)
2021-01-14T23:22:16,104864 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3953, expected 3989)
2021-01-14T23:22:16,126814 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3955, expected 3990)
2021-01-14T23:22:16,146020 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3959, expected 3991)
2021-01-14T23:22:16,205134 13956 0x00007f37c5a21700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 3964, expected 3996)
2021-01-14T23:22:17,294718 13956 0x00007f37e2a5b700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 17574, expected 17580)
2021-01-14T23:22:18,084695 13956 0x00007f3bd1b72700 info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:108 remove_not_supported_codecs_from_array() <kmswebrtcendpoint26784> Removing not supported codec 'AMR/8000'
2021-01-14T23:22:18,084790 13956 0x00007f3bd1b72700 info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:566 WebRtcEndpointImpl() STUN port not found in config; using default value: 3478
2021-01-14T23:22:18,084811 13956 0x00007f3bd1b72700 info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:574 WebRtcEndpointImpl() STUN server not found in config; remember that NAT traversal requires STUN or TURN
2021-01-14T23:22:18,084828 13956 0x00007f3bd1b72700 info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:592 WebRtcEndpointImpl() Using TURN relay server: <user:password>@51.161.12.163:3478?transport=udp
2021-01-14T23:22:18,090658 13956 0x00007f37d9248700 fixme basesink gstbasesink.c:3125 gst_base_sink_default_event() <nicesink26784> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
2021-01-14T23:22:18,090965 13956 0x00007f37901b6700 fixme default gstutils.c:3766 gst_pad_create_stream_id_internal() <nicesrc26784:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
2021-01-14T23:22:18,091357 13956 0x00007f3bcf36d700 info basertpendpoint kmsbasertpendpoint.c:1118 kms_base_rtp_endpoint_start_transport_send() <kmswe
Expected & current behavior
Kurento should be running without crashing, not sure what went wrong.
Info about your environment
Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-128-generic x86_64) Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz cores = 6 enabledcores = 6 threads = 12 System Memory = 32GiB
About Kurento Media Server
Kurento Media Server version: 6.15.0 Found modules: 'core' version 6.15.0 'elements' version 6.15.0 'filters' version 6.15.0
About your Application Server
- Programming Language: Node.js
Run these commands
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
Kurento Media Server version: 6.15.0
ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.18.04 amd64 libav plugin for GStreamer
ii gstreamer1.5-nice:amd64 0.1.18-0kurento1.18.04 amd64 ICE library (GStreamer 1.5 plugin)
ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "bad" set
ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins from the "base" set
ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "good" set
ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "ugly" set
ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugin for PulseAudio
ii kms-core 6.15.0-0kurento1.18.04 amd64 Kurento Core module
ii kms-elements 6.15.0-0kurento1.18.04 amd64 Kurento Elements module
ii kms-filters 6.15.0-0kurento1.18.04 amd64 Kurento Filters module
ii kms-jsonrpc 6.15.0-0kurento1.18.04 amd64 Kurento JSON-RPC library
ii kmsjsoncpp 1.6.3-1kurento1.18.04 amd64 Kurento jsoncpp library
ii kurento-media-server 6.15.0-0kurento1.18.04 amd64 Kurento Media Server
ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer development files for libraries from the "bad" set
ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer libraries from the "base" set
ii libgstreamer1.5-0:amd64 1.8.1-1kurento2.18.04 amd64 Core GStreamer libraries and elements
ii libnice10:amd64 0.1.18-0kurento1.18.04 amd64 ICE library (shared library)
ii libsrtp0:amd64 1.6.0-0kurento1.18.04 amd64 Secure RTP (SRTP) and UST Reference Implementations - shared library
ii libusrsctp 0.9.2-1kurento1.18.04 amd64 sctp-refimpl library
ii openh264 1.5.0-0kurento1.18.04 amd64 H.264 Video Codec provided by Cisco Systems, Inc.
ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from openh264
ii openwebrtc-gst-plugins 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins
Hello @josephmiller2000! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.
To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Ok here is the error.log file details
2021-01-04T08:04:55+00:00 -- New execution
0:00:00.026027216 9077 0x561dc5f9d530 WARN ladspa gstladspa.c:455:plugin_init: no LADSPA plugins found, check LADSPA_PATH
0:00:00.088263179 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.088277899 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.088282197 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.088285782 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.088288959 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.088291937 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.088295381 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.088298954 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.088303212 9077 0x561dc5f9d530 WARN default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
2021-01-04T15:07:45+00:00 -- New execution
2021-01-04T18:43:57+00:00 -- New execution
2021-01-04T18:56:54+00:00 -- New execution
2021-01-05T10:21:18+00:00 -- New execution
2021-01-05T11:18:43+00:00 -- New execution
2021-01-05T11:29:26+00:00 -- New execution
2021-01-05T11:32:11+00:00 -- New execution
2021-01-07T07:59:20+00:00 -- New execution
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!
2021-01-22T07:14:44+00:00 -- New execution
Ok kurento crashed again, without any errors. But this time, i could not start kurento again, until i reboot the server.
Here is the screenshot image, before i stopped kurento, you can check how long it was running without crash.
Here is the error i keep getting if i try to start the kurento, after todays crash.
About the error.log file, here is the details.
2021-01-05T11:29:26+00:00 -- New execution
2021-01-05T11:32:11+00:00 -- New execution
2021-01-07T07:59:20+00:00 -- New execution
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!
2021-01-22T07:14:44+00:00 -- New execution
2021-02-09T04:34:07+00:00 -- New execution
Before trying to start KMS (sudo service kurento-media-server start), check that there are no other KMS instances running by some chance (Check your installation):
$ ps -fC kurento-media-server
UID PID PPID C STIME TTY TIME CMD
Now if it's possible, clean the logs directory (delete or move them somewhere else just during these tests):
sudo service kurento-media-server stop
sudo mv /var/log/kurento-media-server /var/log/kurento-media-server_BACKUP
sudo service kurento-media-server start
And paste here the tail of all log files (this is why I suggest moving out the old logs, so they don't show up here):
tail -n 20 /var/log/kurento-media-server/*.log
About the error.log file, here is the details.
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!
This message Error: the pic_width_in_mbs exceeds the level limits comes from the H.264 parser (OpenH264). It means that the encoder is sending a wrong H.264 stream.
Please provide more info about the H.264 video: where does it come from? Is it a web browser, an IP camera? Are you sending it to Kurento with a WebRtcEndpoint? RtpEndpoint? PlayerEndpoint?
If it is an IP camera or other kind of device under your control: how is the video configuration?
Access your device settings and make sure that H.264 is encoded with Constrained baseline profile and Level 3.1
About the error.log file, here is the details.
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!This message Error: the pic_width_in_mbs exceeds the level limits comes from the H.264 parser (OpenH264). It means that the encoder is sending a wrong H.264 stream.
Please provide more info about the H.264 video: where does it come from? Is it a web browser, an IP camera? Are you sending it to Kurento with a WebRtcEndpoint? RtpEndpoint? PlayerEndpoint?
If it is an IP camera or other kind of device under your control: how is the video configuration?
Access your device settings and make sure that H.264 is encoded with Constrained baseline profile and Level 3.1
Thanks for the comment, but this is different issue i guess.
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!
This error happened on, 2021-01-07T07:59:20+00:00
But the next two execution logs are clean without any errors.
Im using kurento for a public chat room, where users can broadcast their webcam video and rest of the people can see them.
It is a browser based application, and H.264 is sending through browser.