bugtracker KMS7 (bionic-gstreamer), PlayerEndpoint consuing RTSP stream + WebRTCEndpoint: Bad video appearances

https://github.com/Kurento/bugtracker/issues/492

Prerequisites

[x] I agree to fill this issue template.
[x] I have read the Troubleshooting Guide and Support Instructions.

Issue description

As I posted to the mailing list, videos streamed via PlayerEndpoint consuming RTSP have visual problems with the new version of Kurento (bionic-gstreamer). Please see below (note that they have slight delays at starting).

KMS6	KMS7

KMS pipeline is a simple one like PlayerEndpoint (uri=rtsp://....) -> WebRtcEndpoint. Please see the JS client code that controls KMS pipelines.

During this problem, KMS7 outputs error logs like below:

0:04:19.798065219  1246 0x7fad4400e0f0 ERROR                  libav :0:: no frame!
0:04:19.798081648  1246 0x7fad4400e0f0 WARN                   libav gstavviddec.c:1751:gst_ffmpegviddec_frame:<avdec_h264-0> avdec_h264: decoding error (len: -1094995529, have_data: 0)
0:04:19.848197348  1246 0x7fad4400e0f0 ERROR                  libav :0:: no frame!
0:04:19.848259666  1246 0x7fad4400e0f0 WARN                   libav gstavviddec.c:1751:gst_ffmpegviddec_frame:<avdec_h264-0> avdec_h264: decoding error (len: -1094995529, have_data: 0)

However, KMS6 also outputs similar logs like below though it does not have visual problems:

0:04:19.223950365  1381 0x7f8148090770 ERROR                  libav :0:: no frame!
0:04:19.223962425  1381 0x7f8148090770 WARN                   libav gstavviddec.c:1528:gst_ffmpegviddec_frame:<avdec_h264-0> avdec_h264: decoding error (len: -1094995529, have_data: 0)
0:04:19.724015901  1381 0x7f8148090770 ERROR                  libav :0:: no frame!
0:04:19.724028656  1381 0x7f8148090770 WARN                   libav gstavviddec.c:1528:gst_ffmpegviddec_frame:<avdec_h264-0> avdec_h264: decoding error (len: -1094995529, have_data: 0)

I'm using docker and KMS7 is running inside kurento/kurento-media-server-exp:bionic-gstreamer with little modifications and KMS6 is kurento/kurento-media-server:6.14.0 (details below).

(These logs looks same to the one in #492 )

The camera I'm using is SNC-CX600W from SONY. I also found JVC VN-H237B makes a similar problem.

Context

I simply tried to play videos from RTSP streams via KMS7, which had been successful with KMS6.

How to reproduce?

Launch KMS7
- I usually use Docker and Docker Compose with this Dockerfile and this docker-compose.yml
  - The docker image is almost same to kurento/kurento-media-server-exp:bionic-gstreamer but the newest entrypoint.sh is added.
- In my environment, KMS in docker is hosted in a server different from localhost but inside the same LAN witout STUN servers, then KMS_EXTERNAL_ADDRESS is set to the local IP address of the server.
- I also found the HEAD of bionic-gstreamer branch of kms-omni-build can reproduce the bug in the same environment. To reproduce it, I simply add volumes directive to the docker-compose.yml above to mount the kms-omni-build directory to the docker environment.
Create this pipeline: "PlayerEndpoint (uri=rtsp://....) -> WebRtcEndpoint".
- The JS code I'm using is here,
  - This client is constructed upon Create-React-App template. It simply starts yarn start in frontend directory of the repo.
Open the page served by step 2.
Enter RTSP URL to the input.
Click "Start" button and see the error.

A big problem here is that I cannot share the RTSP stream to reproduce the bug, while I can say that I reproduced the bug with SONY SNC-CX600W and JVC VN-H237B. I hope you also reproduce this bug with your environment...

Expected & current behavior

See the images above that describes the expected appearance by KMS6 and the current behavior of KMS7.

(Optional) Possible solution

I found the following workarounds.

Removing `libavformat`

As it seems the error is around libav, I firstly investigated arount it and found the versions of libav are different between kurento/kurento-media-server-exp:bionic-gstreamer (/usr/lib/x86_64-linux-gnu/libavformat.so.57.83.100) and kurento/kurento-media-server:6.14.0 (/usr/lib/x86_64-linux-gnu/libavformat-ffmpeg.so.56.40.101). Then I think the fundamental problem is in it.

When I simply removed libavformat (apt remove libavformat57) and built KMS based on kms-omni-build, PlayerEndpoint used OpenH264 instead and the visual problem disappeared though different errors below came out.

0:01:52.409126035  1553 0x7fc2b8248050 WARN                  vpxenc gstvpxenc.c:1914:gst_vpx_enc_handle_frame:<vp8enc3> Ignoring too small frame duration 0:00:00.000000000
0:01:52.468570387  1553 0x7fc2ac004800 ERROR                   glib (NULL domain):0:: gst_video_codec_frame_unref: assertion 'frame != NULL' failed
0:01:52.609023737  1553 0x7fc2ac004800 ERROR                   glib (NULL domain):0:: gst_video_codec_frame_unref: assertion 'frame != NULL' failed

Of course, removing libav is not an optimal solution, but it may be a good entrypoint to investigate the bug.

Removing `drop-on-latency=TRUE` from `rtspsrc`

When I removed drop-on-latency=TRUE from rtspsrc inside PlayerEndpoint by commenting out this line, the visual problem disappeared though the error logs are still there. It was same to the case of KMS6.

However, I don't know it was a right fix because the option has been introduced on purpose by https://github.com/Kurento/kms-elements/pull/11/files

Switching `uridecodebin` to `uridecodebin3`

When I changed uridecodebin inside PlayerEndpoint to uridecodebin3 by fixing this line, the visual problem disappeared and the error logs are also no longer there.

It looks like a complete solution, but I still don't know it is because uridecodebin3 does not emit element-added signal therefore setting options to rtspsrc inside it is skipped.

INFO about Kurento Media Server

Kurento version: bionic-gstreamer
Server OS: 18.04 (Bionic)
Installation method:
- [ ] apt-get
- [x] Docker
- [ ] AWS CloudFormation
- [x] Built from sources

INFO about your Application Server

Programming Language: browser JavaScript
Kurento Client version: 6.14.0

INFO about end-user clients

Device(s): Mac
OS(es): macOS Catalina (10.15.6)
Browser(s): Chrome 85.0.4183.121

INFO about your environment

In my env, KMS_EXTERNAL_ADDRESS option is used because the server is in the same LAN without STUN servers. See "How to reproduce?" section above for the details.

Run these commands

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

kurento-media-server --version
0:00:00.011651580    54 0x5567fd867560 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so
0:00:00.011687502    54 0x5567fd867560 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: elements, version: 7.0.0~25.gfe11a16, date: Jan 24 2020 18:02:07
0:00:00.012983946    54 0x5567fd867560 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so
0:00:00.013011608    54 0x5567fd867560 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: filters, version: 7.0.0~10.gf2d83e5, date: Jan 24 2020 18:12:06
0:00:00.013315027    54 0x5567fd867560 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so
0:00:00.013343891    54 0x5567fd867560 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: core, version: 7.0.0~10.g1f1ce74d, date: Jan 24 2020 17:54:12
Kurento Media Server version: 7.0.0~15.g692a6f5
Found modules:
        'core' version 7.0.0~10.g1f1ce74d
        'elements' version 7.0.0~25.gfe11a16
        'filters' version 7.0.0~10.gf2d83e5

dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.0-nice:amd64              0.1.16-0kurento1.18.04~20200124183407.gbpd3ba5e           amd64        ICE library (GStreamer plugin)
ii  gstreamer1.0-nice-dbgsym:amd64       0.1.16-0kurento1.18.04~20200124183407.gbpd3ba5e           amd64        debug symbols for gstreamer1.0-nice
ii  kms-core                             7.0.0-0kurento1.18.04~20200124183407.gbp1f1ce7            amd64        Kurento Core module
ii  kms-core-dbg                         7.0.0-0kurento1.18.04~20200124183407.gbp1f1ce7            amd64        Kurento Core module
ii  kms-elements                         7.0.0-0kurento1.18.04~20200124183407.gbpfe11a1            amd64        Kurento Elements module
ii  kms-elements-dbg                     7.0.0-0kurento1.18.04~20200124183407.gbpfe11a1            amd64        Kurento Elements module
ii  kms-filters                          7.0.0-0kurento1.18.04~20200124183407.gbpf2d83e            amd64        Kurento Filters module
ii  kms-filters-dbg                      7.0.0-0kurento1.18.04~20200124183407.gbpf2d83e            amd64        Kurento Filters module
ii  kms-jsonrpc                          7.0.0-0kurento1.18.04~20200124183407.gbpe7f3e5            amd64        Kurento JSON-RPC library
ii  kms-jsonrpc-dbg                      7.0.0-0kurento1.18.04~20200124183407.gbpe7f3e5            amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                           1.6.3-1kurento1.18.04~20200124183407.gbp15940b            amd64        Kurento jsoncpp library
ii  kmsjsoncpp-dbg                       1.6.3-1kurento1.18.04~20200124183407.gbp15940b            amd64        Kurento jsoncpp library
ii  kurento-dbg                          7.0.0-0kurento1.18.04~20200124183407.gbp692a6f            amd64        Meta-package that installs debug symbols
ii  kurento-media-server                 7.0.0-0kurento1.18.04~20200124183407.gbp692a6f            amd64        Kurento Media Server
ii  kurento-media-server-dbg             7.0.0-0kurento1.18.04~20200124183407.gbp692a6f            amd64        Kurento Media Server
ii  libnice10:amd64                      0.1.16-0kurento1.18.04~20200124183407.gbpd3ba5e           amd64        ICE library (shared library)
ii  libnice10-dbgsym:amd64               0.1.16-0kurento1.18.04~20200124183407.gbpd3ba5e           amd64        debug symbols for libnice10
ii  libsrtp2-1:amd64                     1:2.1.0-1kurento1.18.04~20200124183407.gbp2061ec          amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp1                          0.9.3.0+20190509-0kurento1.18.04~20200124183407.gbp85d78a amd64        portable SCTP userland stack - shared library
ii  openh264                             1.8.0-0kurento1.18.04~20200124183407.gbpe251c8            amd64        OpenH264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugin                  1.0.0-0kurento1.18.04~20200124183407.gbp92ba0d            amd64        OpenH264 plugin for GStreamer
ii  openwebrtc-gst-plugins               0.10.0+20190509-0kurento1.18.04~20200124183407.gbp87c96f  amd64        OpenWebRTC specific GStreamer plugins
ii  openwebrtc-gst-plugins-dbg           0.10.0+20190509-0kurento1.18.04~20200124183407.gbp87c96f  amd64        OpenWebRTC specific GStreamer plugins

Oct 01 '20 02:10 whitphx

Could you please try using bigger and lower networkCache? (RTSP broken video) The default is 2000 milliseconds. Try something like 100, and also try 10000. It wouldn't make sense that a shorter buffer solves issues, but curiously enough some users have reported so in the past, so it's worth a try.

Using drop-on-latency=TRUE means that incoming frames are dropped when the buffer is full. Before this, without drop-on-latency=TRUE what happened is that the internal buffer would grow bigger and bigger, eventually consuming all of the machine's memory. This happens when the CPU is lagging behind and not able to consume data fast enough to prevent the buffer from filling. It might make sense that your issue disappears when removing drop-on-latency=TRUE, but in that case you'll probably see your memory usage grow over time.

Maybe we should update the default buffer, so it is bigger by default. I'm open to discussing this.

Re: OpenH264, it is interesting that this problem doesn't happen with this decoder. Maybe it works much faster than libav? Although the errors further down the line in the Pipeline make me think that the issue is still there, but just shows up in a different way.

As it seems the error is around libav, I firstly investigated arount it and found the versions of libav are different between kurento/kurento-media-server-exp:bionic-gstreamer (/usr/lib/x86_64-linux-gnu/libavformat.so.57.83.100) and kurento/kurento-media-server:6.14.0 (/usr/lib/x86_64-linux-gnu/libavformat-ffmpeg.so.56.40.101).

This happens because kurento/kurento-media-server:6.14.0 uses the GStreamer 1.8 fork on Ubuntu 16.04, and kurento/kurento-media-server-exp:bionic-gstreamer should be using the system's official GStreamer build on Ubuntu 18.04, so it makes sense that libraries are different versions in these two images.

Oct 02 '20 08:10 j1elo

Thank you for the suggestion, but though I tried 10, 100, and 10000 for networdCache option, none of them worked.

What do you think about trying to use uridecodebin3 with drop-on-latency=TRUE option? As I have shown, with uridecodebin3, the error didn't occur somehow. It seems it worked correctly on decoding. Then, it may be successful even with drop-on-latency=TRUE.

Oct 05 '20 05:10 whitphx

I tried it and it seemed to work. I will create a PR as a next starting point for discussion.

EDIT: The PR introducing uridecodebin3 fixes the problem and properly configure rtspsrc inside it.

Oct 05 '20 06:10 whitphx

bugtracker bugtracker copied to clipboard

KMS7 (bionic-gstreamer), PlayerEndpoint consuing RTSP stream + WebRTCEndpoint: Bad video appearances

Prerequisites

Issue description

Context

How to reproduce?

Expected & current behavior

(Optional) Possible solution

Removing libavformat

Removing drop-on-latency=TRUE from rtspsrc

Switching uridecodebin to uridecodebin3

INFO about Kurento Media Server

INFO about your Application Server

INFO about end-user clients

INFO about your environment

Run these commands

bugtracker
bugtracker copied to clipboard

Removing `libavformat`

Removing `drop-on-latency=TRUE` from `rtspsrc`

Switching `uridecodebin` to `uridecodebin3`