Black bars and wrong aspect ratio when using videocrop on composite inputs

[repomaa]

Prerequisites

• [X] I have read the SUPPORT document

• [X] I have checked the Troubleshooting Guide

• [X] I have tested with the latest version of Kurento

Issue description

Adding videobox or videocrop gstreamer filters between webrtc peer connections and a composite hub port does not work correctly.

Context

I'm trying to build a 3x3 grid from 9 incoming webrtc video streams. I created a single pipeline, created a Composite hub and for each incoming webrtc peer connection i create a gstreamer filter that crops the video to have a 1:1 aspect ratio. then i connect the peer connection with that filter and the filter with a newly created hubPort for the composite hub. The streams in the resulting video when merged through the Composite hub are all in the wrong aspect ratio (not 1:1, but in something like 3:4 or 9:16). I double checked the crop params and they are correct. After inspecting the dot graph of the pipeline I could see that there's a videoscale happening before the crop which messes up the result. It scales up the video from 960x720 to 1080x1080. It happens in a kmsagnosticbin2 inside kmswebrtcendpoint. I can't tell why it happens because I'm not explicitly scaling anything anywhere. It also seems that adding or removing sources to the Composite hub while it's running changes the aspect ratio of the individual tiles.

How to reproduce?

Create a pipeline WebRTCEndpoint -> GstreamerFilter "videobox top=0 bottom=0 left=120 right=120" - > HubPort of Composite Connect your webrtc client with a camera using a resolution of 920x720 or change the left and right values above to (width - height) / 2

Expected & current behavior

No black bars should be visible and the mixed video streams should all be of 1:1 aspect ratio. Connecting additional streams should not affect the aspect ratio of existing streams in the Composite.

Instead there are black bars and additional streams affect the aspect ratio of existing streams in the Composite hub.

(Optional) Possible solution

Possibly solved by fixing #485

INFO about Kurento Media Server

• Kurento version: nightly
• Server OS: Ubuntu 16.04 (Xenial)
• Installation method:
  • [] apt-get
  • [x] Docker
  • [] AWS
  • [] Built from sources

INFO about your Application Server

• Language: Java
• Kurento Client version: nightly

INFO about end-user clients

• Device(s): PC
• OS(es): ArchLinux
• Browser(s): Firefox Developer Edition (latest)

INFO about your environment

The application isn't deployed. Just runs on local machine for testing.

Commands output

Skipping because of Docker setup (using kurento/kurento-media-server-dev image)

[nightwolftamb]

I have something similar problem some time ago and the reson was that webrtc is sending stream not in exact needed resolution. For example if you require camera 1280x720 - webrtc can rescale it to 640x360. In that case crop will be wrong. I fix my problem by adding capsfilter before crop

[repomaa]

@nightwolftamb cool! Can you show me an example? What happens if the requirements of the capsfilter aren't met?

[repomaa]

@nightwolftamb also: how do you determine the crop? Currently I'm sending the video dimensions from the client but there has to be another way, right?

[j1elo]

The videoscale that happens in kmsagnosticbin is probably there because incoming videos can be in a variety of aspect ratio formats, and resolutions, so a pre-process step must be done before compositing all into a grid.

Composite is written to be an all-in-one module than can be directly provided with output from other Endpoints (WebRtcEndpoint, RtpEndpoint, PlayerEndpoint), so you shouldn't need to add your own pre-process step with scaling. Just provide all your inputs to Composite and it will do whatever is necessary. But of course if you don't like the result, then you'll have to tinker with the Composite code to modify it to your liking.

FYI the actual compositing of video into a grid is done by GStreamer's compositor plugin.

Please let me know if this issue can be closed or you still consider there is an actual bug.