Audio mixed in to Composite Hub is choppy

[joe-at-siemens]

KMS Version:

Kurento Media Server version: 6.11.0 Found modules: 'core' version 6.11.0 'elements' version 6.11.0 'filters' version 6.11.0

Ubuntu Version

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial

Other libraries versions:

ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.16.04 amd64 libav plugin for GStreamer ii gstreamer1.5-nice:amd64 0.1.15-1kurento3.16.04 amd64 ICE library (GStreamer 1.5 plugin) ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento3.16.04 amd64 GStreamer plugins from the "bad" set ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento1.16.04 amd64 GStreamer plugins from the "base" set ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer plugins from the "good" set ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.16.04 amd64 GStreamer plugins from the "ugly" set ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer plugin for PulseAudio ii kms-core 6.11.0-0kurento1.16.04 amd64 Kurento Core module ii kms-elements 6.11.0-0kurento1.16.04 amd64 Kurento Elements module ii kms-filters 6.11.0-0kurento1.16.04 amd64 Kurento Filters module ii kms-jsonrpc 6.11.0-0kurento1.16.04 amd64 Kurento JSON-RPC library ii kmsjsoncpp 1.6.3-1kurento1.16.04 amd64 Kurento jsoncpp library ii kurento-media-server 6.11.0-0kurento1.16.04 amd64 Kurento Media Server ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento3.16.04 amd64 GStreamer development files for libraries from the "bad" set ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento1.16.04 amd64 GStreamer libraries from the "base" set ii libgstreamer1.5-0:amd64 1.8.1-1kurento1.16.04 amd64 Core GStreamer libraries and elements ii libnice10:amd64 0.1.15-1kurento3.16.04 amd64 ICE library (shared library) ii libusrsctp 0.9.2-1kurento1.16.04 amd64 sctp-refimpl library ii openh264 1.4.0-1kurento1.16.04 amd64 OpenH264 library ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento3.16.04 amd64 GStreamer plugins from openh264 ii openwebrtc-gst-plugins 0.10.0-1kurento1.16.04 amd64 OpenWebRTC specific GStreamer plugins

Client libraries

• Language: Node.js
• Version: v9.11.2

Browsers tested

Add OK or FAIL, along with the version, after browsers where you have tested this issue:

• Chrome: FAIL
• Firefox: FAIL

System description:

AWS EC2 instance using TURN

What steps will reproduce the problem?

Connect "VIDEO" from WebRtcEndpoints and PlayerEndpoints to Composite Hub. Connect "AUDIO" from a WebRtcEndpoint.

What is the expected result?

Output from Composite Hub to be grid of the videos mixed together with the audio from the WebRtcEndpoint cleanly and clearly mixed in.

What happens instead?

Video is laid out as expected, but audio is choppy.

Does it happen with one of the tutorials?

Unknown

Please provide any additional information below

Using the latest Docker image:

kurento/kurento-media-server:6.11.0

[not404-found]

I have the same issue but for passThrough. Its very critical

[j1elo]

The audio composite plugin from GStreamer is not of the best quality... however there are some other users of Composite that didn't have any issues. I'd check if disabling browser filtering (noise reduction, etc.) helps at all. If still happens, please write a minimum application that can be used to reproduce the issue.

[j1elo]

@not404-found we are currently focused on performance, resource usage, and other stability improvements for Kurento. If this issue is very critical for you, I recommend you to contract support services: https://doc-kurento.readthedocs.io/en/latest/user/support.html#commercial-support

[joe-at-siemens]

I just created a minimal test script and environment using the latest Docker image (6.11.0) running on Windows. I use 2 PlayerEndpoints playing .webm files obtained from webmfiles.org. It starts by connecting one PlayerEndpoint to a RecorderEndpoint (via PassThrough), then reconfigures the pipeline to use a Composite Hub. It then adds the second PlayerEndpoint, before disconnecting the Composite and plugging the second PlayerEndpoint into the RecorderEndpoint.

const kurento = require('kurento-client');

const sleep = (milliseconds) => {
    return new Promise(resolve => setTimeout(resolve, milliseconds))
  }  

kurento('ws://127.0.0.1:8888/kurento')
    .then(client => {
        return client.create('MediaPipeline')
            .then(pipeline => {
                return {
                    pipeline: pipeline,
                    client: client
                }
            })
    })
    .then(async result => {
        let pipeline = result.pipeline;
        let client = result.client;

        let playerEndpoint1 = await pipeline.create('PlayerEndpoint', {uri: 'file:///world/bunny.webm'});        
        await playerEndpoint1.play();

        let passthrough = await pipeline.create('PassThrough');
        let recorder = await pipeline.create('RecorderEndpoint', {uri: 'file:///world/output.webm', mediaProfile: 'WEBM_AUDIO_ONLY'})
        await passthrough.connect(recorder, "AUDIO");


        await playerEndpoint1.connect(passthrough, "AUDIO");
        await recorder.record();
        await sleep(7500)

        /// Create Composite Hub and hub ports
        let composite = await pipeline.create('Composite')
        let hub1 = await composite.createHubPort();
        let hub2 = await composite.createHubPort();
        let hub3 = await composite.createHubPort();

        await playerEndpoint1.disconnect(passthrough, "AUDIO");
        await playerEndpoint1.connect(hub1, "AUDIO");
        await hub3.connect(passthrough, "AUDIO")
        await sleep(7500)

        /// Create and connect second input
        let playerEndpoint2 = await pipeline.create('PlayerEndpoint', {uri: 'file:///world/elephants-dream.webm'});
        await playerEndpoint2.play();
        await playerEndpoint2.connect(hub2, "AUDIO")

        await sleep(7500);

        /// Disconnect and release
        await playerEndpoint1.disconnect(hub1, "AUDIO");
        await playerEndpoint2.disconnect(hub2, "AUDIO");
        await hub3.disconnect(passthrough, "AUDIO");
        await playerEndpoint1.stop();
        await playerEndpoint1.release();
        await Promise.all([hub1, hub2, hub3].map(obj => obj.release()))
        await composite.release();

        await playerEndpoint2.connect(passthrough, "AUDIO");
        await sleep(15000);

        /// stop and cleanup
        await recorder.stopAndWait();
        await pipeline.release();

        /// stop client
        await client.close();        
    })
    .then(() => {
        console.log('finished')
        process.exit(0);
    })

I don't detect the choppiness, clicks or pops I observed in my full-blown solution. My theory is that it is something to do with how queues fill up when the server is under serious load. Audio file can be obtained from this archive output.zip. Unfortunately I have run out of time to work on this issue, but I shall eagerly follow others' feedback.

[Diabl0Man]

I have the same issue, I detected choppiness only when connect video and audio. If I connect only audio then there are no choppiness.

[joe-at-siemens]

I did some extra experiments using Docker Desktop for Windows with different numbers of CPU cores. Using PlayerEndpoints in a MediaPipeline under extremely high loads I was able to reproduce some audio degradation. However, I don't think this is the cause of the choppiness I observed in my original system. I suspect it has something to do with video sources using WebRtcEndpoint, and may not necessarily be a problem caused directly by the Composite Hub.