Kurento
'MediaFlowInStateChange' event is not triggered for Node js
Prerequisites
These are MANDATORY, otherwise the issue will be automatically closed.
- [x] I agree to fill this issue template.
- [x] I have read the Troubleshooting Guide and Support Instructions.
Issue description
I am getting 0 size recording file once in 10 times, i have read the trouble shooting guide and followed all the points but -
As stated in documentation, that recording should be started when -
RecorderEndpoint has notified a MediaFlowInStateChange for ALL streams (so, if you record AUDIO+VIDEO, your application must receive a MediaFlowInStateChange event for audio, and another MediaFlowInStateChange event for video).
But 'MediaFlowInStateChange' is not triggered for recorder endpoint. But 'MediaFlowOutStateChange' is triggered for webrtc endpoint.
Webrtc endpoint is connected with Recorder endpoint.
Sample code -
let recorderVideo = false;
let recorderAudio = false;
recorderEndpoint.on('MediaFlowInStateChange', function(event){
console.log('recorderEndpoint MediaFlowInStateChange ', event);
if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
recorderAudio=true;
}
if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
recorderVideo=true;
}
if(recorderAudio && recorderVideo){
recorderEndpoint.record((err) => {
if (err) {
console.error('error when trying to record');
pipeline.release();
callback(err);
}
});
}
});
Recording never starts as 'MediaFlowInStateChange' is not triggered for recorder Endpoint
Context
Need to record video from browser in Safari, as Media Recorder API is not yet supported by default in Safari, So using webrtc for the same
How to reproduce?
I changed Magic-Mirror-Node Tutorial to support recording by replacing -
faceOverlayFilter pipeline to RecorderEndpoint pipeline
And to start recording I added below code to check if 'MediaFlowInStateChange' is flowing for both Audio and Video for Recorder Endpoint, and only then start recording but this event is not fired.
let recorderVideo = false;
let recorderAudio = false;
recorderEndpoint.on('MediaFlowInStateChange', function(event){
console.log('recorderEndpoint MediaFlowInStateChange ', event);
if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
recorderAudio=true;
}
if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
recorderVideo=true;
}
if(recorderAudio && recorderVideo){
recorderEndpoint.record((err) => {
if (err) {
console.error('error when trying to record');
pipeline.release();
callback(err);
}
});
}
});
Expected & current behavior
Expected behavior - 'MediaFlowInStateChange' should have been fired in recorder endpoint as 'MediaFlowOutStateChange' for webrtc endpoint was fired for both audio and video.
Current behavior - 'MediaFlowInStateChange' is not fired in recorder endpoint even when 'MediaFlowOutStateChange' for webrtc endpoint was fired for both audio and video.
(Optional) Possible solution
Info about your environment
About the application -
- I am using Webrtc Endpoint which is connected to Recorder Endpoint
- Coturn server is installed and configured with KMS
About Kurento Media Server
- Kurento version: 6.15.0
- Server OS: 18.04 (Bionic)
- Installation method:
- [x] apt-get
About your Application Server
- Programming Language: Node.js
- Kurento Client version: 6.15.0
About end-user clients
- Device(s): MacBook Pro (Mid 2014 model - 13 inch)
- OS(es): macOS Catalina
- Browser(s): Safari version 14.0
Run these commands
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
Kurento Media Server version: 6.15.0
Found modules:
'core' version 6.15.0
'elements' version 6.15.0
'filters' version 6.15.0
ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.18.04 amd64 libav plugin for GStreamer
ii gstreamer1.5-nice:amd64 0.1.18-0kurento1.18.04 amd64 ICE library (GStreamer 1.5 plugin)
ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "bad" set
ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins from the "base" set
ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "good" set
ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "ugly" set
ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugin for PulseAudio
ii kms-core 6.15.0-0kurento1.18.04 amd64 Kurento Core module
ii kms-elements 6.15.0-0kurento1.18.04 amd64 Kurento Elements module
ii kms-filters 6.15.0-0kurento1.18.04 amd64 Kurento Filters module
ii kms-jsonrpc 6.15.0-0kurento1.18.04 amd64 Kurento JSON-RPC library
ii kmsjsoncpp 1.6.3-1kurento1.18.04 amd64 Kurento jsoncpp library
ii kurento-media-server 6.15.0-0kurento1.18.04 amd64 Kurento Media Server
ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer development files for libraries from the "bad" set
ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer libraries from the "base" set
ii libgstreamer1.5-0:amd64 1.8.1-1kurento2.18.04 amd64 Core GStreamer libraries and elements
ii libnice10:amd64 0.1.18-0kurento1.18.04 amd64 ICE library (shared library)
ii libsrtp0:amd64 1.6.0-0kurento1.18.04 amd64 Secure RTP (SRTP) and UST Reference Implementations - shared library
ii libusrsctp 0.9.2-1kurento1.18.04 amd64 sctp-refimpl library
ii openh264 1.5.0-0kurento1.18.04 amd64 H.264 Video Codec provided by Cisco Systems, Inc.
ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from openh264
ii openwebrtc-gst-plugins 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins
Hello @codercoder12! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.
To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi, does the Kurento logs show any suspicious Warning or Error message?
does this issue happen randomly? You said it fails 1 in each 10 times... is there any difference between that failed time and the successful ones? (differences in devices, browsers, number of peers connected to the same Pipeline, different Pipeline, different recording format... anything you can think of)
is the WebRtcEndpoint connected directly to the Recorder? Or are there any other intermediate elements in the Pipeline?
Are you getting the Error event from the Pipeline itself, or any of the Elements/Endpoints inside?
-- does the Kurento logs show any suspicious Warning or Error message? --> log says that Error: Already in state stop
--does this issue happen randomly? You said it fails 1 in each 10 times. --> Yes it fails(by failing I meant i get 0 size recorded files) randomly 1 in 10 times meant 10-20 % of the time I get 0 size recorded files, but sometimes it is 30-40%
--is there any difference between that failed time and the successful ones? (differences in devices, browsers, number of peers connected to the same Pipeline, different Pipeline, different recording format... anything you can think of). --> No, no difference at all, same safari browser(14.0) in same macbook pro, only one peer connected to the piepline (which is me, as i am testing it),same recording format which is MKV(as our requirement is to use H264 and AAC).
Only webrtc endpoint is connected with recorder endpoint(As i want to record video in browser from safari, as media recorder api is not suported by default in safari)
Everything is same, it is just that I was getting 0 size recorded files, then I read the trouble shooting guide to remove 0 size recorded files. In it, it is mentioned to wait for MediaFlowInStateChange for ALL streams (so, if you record AUDIO+VIDEO, your application must receive a MediaFlowInStateChange event for audio, and another MediaFlowInStateChange event for video) for recorder Endpoint, only then start recording.
But MediaFlowInStateChange event is never triggered in node, so recorder endpoint never starts recording with below code--
let recorderVideo = false;
let recorderAudio = false;
recorderEndpoint.on('MediaFlowInStateChange', function(event){
console.log('recorderEndpoint MediaFlowInStateChange ', event);
if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
recorderAudio=true;
}
if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
recorderVideo=true;
}
if(recorderAudio && recorderVideo){
recorderEndpoint.record((err) => {
if (err) {
console.error('error when trying to record');
pipeline.release();
callback(err);
}
});
}
});
I was using MediaFlowInStateChange event to remove 0 size recorded files, but as it is not fired, recording dosent starts.
But webrtc Endpoint fires MediaFlowOutStateChange event for both Audio and Video
Hi, thanks for the info.
Please let me see your SDP Offer and Answers. For this, first configure Kurento debug logs to this value:
GST_DEBUG="3,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4,KurentoMediaElementImpl:5,agnosticbin*:5,kmssdpsession:5,webrtcendpoint:5,kmswebrtcsession:5,kmsiceniceagent:5"
Then restart Kurento, repeat your test until it fails, and share the logs with us (you can post it in pastebin or similar and share the link here, or send it to the email in my Github profile).
I'm not sure if the previous message got a wrong formatting. Use this value for the debug verbosity:
GST_DEBUG="3,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4,KurentoMediaElementImpl:5,agnosticbin*:5,kmssdpsession:5,webrtcendpoint:5,kmswebrtcsession:5,kmsiceniceagent:5"
Hi, I have shared the logs and error with you on your email mentioned on github profile. Please let me know if anything else is required.
I see this in the logs:
2021-01-15T06:36:29 Kurento Media Server started
2021-01-15T06:41:35 Process SDP Offer 1
2021-01-15T06:41:48 Process SDP Offer 2
2021-01-15T06:42:00 Process SDP Offer 3
2021-01-15T06:42:17 Process SDP Offer 4 NO FLOW AT ALL
Is it correct that the problem happened on attempt 4? Or the attempt 4 was just aborted (you hit Ctrl+C or similar to early-finish execution)?
Yes, it failed on 4th attempt(I clicked stop- to stop the recording and it created 0 size file) and I did not click Ctrl+C or similar to stop early-finish execution
And this was my attempt without using below code(checking for 'MediaFlowInStateChange' events for audio and video)
let recorderVideo = false;
let recorderAudio = false;
recorderEndpoint.on('MediaFlowInStateChange', function(event){
console.log('recorderEndpoint MediaFlowInStateChange ', event);
if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
recorderAudio=true;
}
if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
recorderVideo=true;
}
if(recorderAudio && recorderVideo){
recorderEndpoint.record((err) => {
if (err) {
console.error('error when trying to record');
pipeline.release();
callback(err);
}
});
}
});
I was just using the below code to start the recording without checking the 'MediaFlowInStateChange' in recorderEndpoint, if I use the above code(checking for 'MediaFlowInStateChange' events for audio and video) then recording dosent start, as event is not fired for 'MediaFlowInStateChange' for recorder endpoint.
recorderEndpoint.record((err) => {
if (err) {
console.error('error when trying to record');
pipeline.release();
callback(err);
}
});
What I noticed is that 'MediaFlowInStateChange' events for audio and video for recorder endpoint are fired only when I call recorderEndpoint.record
