KMS 6.61, GST 1.18.4 - RecorderEndpoint refuses to record H.264 to MP4 or MKV

[neilyoung]

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

• [x] I agree to fill this issue template.
• [x] I have read the Troubleshooting Guide and Support Instructions.

Issue description

The browser negotiates a H.264 connection successfully with KMS. After connecting a RecorderEndpoint to the WebRtcEndpoint the recording session is closed several seconds later.

Context

• Browser let's KMS offer
• Agrees on H.264. WebRtcEndpoint receives MediaTranscodingStateChange as NOT_TRANSCODING
• Connecting prepared RecorderEnpoint (to local file, MP4_VIDEO_ONLY) on MediaFlowOutStateChange of WebRtcEndpoint.
• RecorderEndpoint MediaFlowInStateChanged fires with FLOWING
• RecorderEnpoint receives Recording event
• The trace below appears in the KMS log 15 seconds later
• RecordingEndpoint receives NOT_FLOWING event and is stopped then

The log output:

2022-11-23T15:31:48,375346 977 0x0000ffff07ffe690 warning enctreebin                kmsenctreebin.c:458 kms_enc_tree_bin_configure() <kmsenctreebin32>  Invalid encoder for caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, width=(int)640, height=(int)360, codec_data=(buffer)0142c01fffe1000f6742c01f8c8d405017fcb00f08846a01000468ce3c80, level=(string)3.1, profile=(string)constrained-baseline, framerate=(fraction)0/1, parsed=(boolean)true
2022-11-23T15:31:50,358623 977 0x0000fffeeb7fd690 warning recorderendpoint          kmsrecorderendpoint.c:1346 kms_recorder_endpoint_on_eos() <kmsrecorderendpoint40>  Releasing pending pads

How to reproduce?

Hmm. Not really an idea

Expected & current behavior

Should record to MP4 as it does if the video connection runs VP8

Info about your environment

• KMS source compilation, GST source compilation. Running in VM on same machine

About Kurento Media Server

• Kurento version: 6.16.0
• Server OS: Ubuntu 20.04 ARM64
• Installation method:
  • [] apt-get
  • [] Docker
  • [] AWS CloudFormation
  • [x] Built from sources

About your Application Server

• Programming Language: Node JS
• Kurento Client version: 6.16.0

About end-user clients

• Device(s): Mac
• OS(es): Ventura 13.01
• Browser(s): Chrome 107.0.5

Run these commands

cat /etc/lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

kurento-media-server --version

Kurento Media Server version: 6.16.0
Found modules:
	'core' version 6.16.1~15.gcf111ef4
	'elements' version 6.16.1~19.gb38df49
	'filters' version 6.16.1~3.gec9da10
	'webrtcarucocode' version 0.1.6~14.g7bb9038
	'webrtcbarcode' version 0.1.6~5.gfd167d4
	'webrtccbr' version 0.1.6~7.gb9ddf5e

dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'

no output

[github-actions[bot]]

Hello @neilyoung! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.

To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.

[neilyoung]

Additional observation: If the browser/KMS connection runs VP8, the recording starts. But the recorded file does not follow the resolution changes from 640x360 over 960x540 to 1280x720, but is stuck at 640x360 seemingly. And it contains an intra frame every 5 seconds as it seems.

[neilyoung]

And - as with the recent RTSP client/server issue: The a.m. KMS log does NOT appear, if I feed the KMS with a constant resolution H.264 stream, coming e.g. from a drone. Perfect video as MP4. No transcoding. No resolution change. But most likely also no flexibility for changing conditions. The encoder on the other side is GStreamer v4l2h264enc.

Does that all make sense?

[neilyoung]

Would it be possible to have a comment? I mean it is impossible to record H.246 to MP4 or MKV when the input stream changes resolution. Nobody concerned?

[j1elo]

This might be something similar to what we found with GStreamer 1.18, while developing Kurento 7.0: https://github.com/Kurento/bugtracker/issues/535

The issue was that, at some point, GStreamer stopped allowing renegotiation of resolution, thus it would halt the stream when one of those changes happened.

The ancient version of GStreamer 1.8 (which is what Kurento 6.x uses) didn't have this limitation, however according to your comments, it might be that the limitation didn't exist for WEBM, but it did actually exist for MP4 and MKV (albeit that'd surprise me because the MKV and WEBM codebases are practically the same).

You commented here https://github.com/Kurento/bugtracker/issues/484#issuecomment-1332688319 that this is precisely the issue. If that's the case, I guess the solution would be to see if GStreamer ended up adding more flexibility to the caps renegotiation, to allow for resolution changes, such as what they did for WEBM/Matroska as I linked here https://github.com/Kurento/bugtracker/issues/535#issuecomment-1280626392

[j1elo]

Yeah, I found that something similar happened to the MP4 muxer, and they added a patch to allow resolution changes, around last year:

• https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/1063
• https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/1071

I'll do some tests and see if a similar patch is needed and can be integrated for Kurento 7.0, then look into backporting it to 6.x

[neilyoung]

Yes, I agree. This is definitely a GStreamer qtmux thing.

For me it is now no longer an issue, because I completely dropped using KMS RecorderEndpoint (which relies on GStreamer). Instead I'm - depending on the video format of the publisher WebRtc stream - negotiating either a local VP8 or H.264 session with the publisher WebRtcEndpoint. Those RTP endpoints then internally feed spawned FFMPEG processes, which do the conversion to either MP4 or WEBM. Works perfectly, follows each caps change.