EricssonResearch
Received video displays a lot of black frames
When receiving video, the video that gets displayed seems to work somewhat OK but has intermittent bursts of displaying black frames and seems to be quite sensitive. More sensitive than it should be.
We noticed that this happens at the same time as RTP elements report a clock skew recalculation.
The black frames come from intervideosrc and happen when frames are not arriving in time, or there is a synchronization problem in or before intervideosink. If there's skew detected in RTP, that's a bad sign, especially between two Owr clients.
What are you testing here? What's your test case? Which clients, etc
Nothing special really. It's happening very reproducibly with the daemon and Safari vs Chrome/Firefox but also between native applications on iOS and chrome/Firefox. Only the remote view is exhibiting this behaviour. Two way video (audio or not doesn't seem to have any impact).
Oh, we were first testing on the same machine via localhost but also saw the same between two different os x machines.
Seems like symptoms identical to https://github.com/EricssonResearch/openwebrtc/issues/186, https://github.com/EricssonResearch/openwebrtc/issues/260, https://github.com/EricssonResearch/openwebrtc/issues/244.
Ok, that's definitely an RTP/transport related problem then... and most likely unrelated to anything else in the pipeline. Maybe related to the changes I started to only start sending data when we actually set up the connections.
Just tested with NativeDemo.. received video on iOS is unusable due to blackening out, so I can confirm this. However, received video on Chrome end is looking pretty good now.
I'd really like to see this issue solved..
(By the way, this is with libvpx's error concealment enabled. Obviously doesn't make a difference.)
@superdump Just tested with latest OpenWebRTC on iOS <-> Chrome (VP8 codec, video-only @ demo.openwebrtc.io) (http://static.verkstad.net/OpenWebRTC_0.1.3.zip) and noticed that the video eventually dropped out permanently after 15 minutes of calling and the console kept getting spammed with errors with increasing packet #'s:
0:15:21.544332000 511 0x153ae638 WARN rtpjitterbuffer gstrtpjitterbuffer.c:2456:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer0> Packet #9015 too late as #41783 was already popped, dropping
0:15:21.548766000 511 0x153ae638 WARN rtpjitterbuffer gstrtpjitterbuffer.c:2456:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer0> Packet #9016 too late as #41783 was already popped, dropping
Additionally, I've had good mileage with inducing problems in my WiFi network by changing the channels on the router during a call. The network connection is momentarily lost, and rtpjitterbuffer doesn't really seem to be able to fix itself after this happens: video will come through corrupted and rtpjitterbuffer keeps emitting warnings every now and then.
The
0:00:00.153968929 16517 0x204dd90 WARN srtpdec gstsrtpdec.c:1151:gst_srtp_dec_chain:<srtp-decoder> Invalid buffer, dropping
warnings come from receiving packets before the DTLS handshake is finished btw. Should probably just drop packets at the encoder in that situation.
Just before the encoder you mean? A valve that is opened when DTLS is done and the keys are set on srtp stuff?
Yeah, or waiting until my changes where we only start producing data when dtls is done
So, the other problems here are: a) we only allow RTCP every 5 seconds (rtcp-min-interval property default value). That's wrong, and we should implement that properly for the AVPF profiles b) we actually lose packets over loopback, and ask for retransmissions multiple times per second... and also for keyframes
b) is very odd and the fact that we're seeing it on multiple machines is concerning. Is the data flow fine until nicesink? Could it be something with nicesink?
a) Completely agree.
The following is not a fix at all as we need to support more regular feedback for RTX (Generic NACK) and PLI/FIR and more regular feedback in general is good but, we may perhaps let all packets through videorepair instead of, on packet loss, dropping all packets until the next intra. Then decoders can attempt to do error concealment. https://github.com/superdump/openwebrtc/commit/73fb81b6df5003bd60a29ae52a3473f2f0565db0 It would probably have hidden this black frames issue unfortunately, but at the same time, it would probably have hidden this black frames issue! :smile: I'm tempted to not drop until intra from now on and try to instead see when error concealment or the decoder barfs and outputs broken frames and instead try to see if we can improve those cases.
b) is only caused by RTX being broken, I'm fixing that currently :)
For your workaround, I think it makes sense to drop frames until the first keyframe, but after that not anymore.
@sdroege you mean that 1/100 packets are not getting lost on localhost, but rather RTX is causing the session to decide that those packets are lost when they are not actually?
Hello, I am also experiencing the same blackout issue in video transmission on iOS. Are you guys working to fix this? Any ETA for the fix?
We are actively working on this issue. As a consequence of it, we have identified a bunch of issues in GStreamer's RTP/RTCP stack that need fixing. Those are in progress.
We literally just landed a change which improves the situation to a usable state for now. The proper fix will come in a later release.
Removed from the release milestone. We have reduced the RTCP interval to 1s from 5s and disabled RTP retransmission as a temporary workaround. Real fixes will be coming to GStreamer soon.
@superdump Wanted to ask about "Real fixes will be coming to GStreamer soon". Is there any update on that?
Many thanks!
@iivalchev - those are in cerbero master already and will be in the next release when we get to that.
I've been recently trying to use OpenWebRTC in Google Glass on one end and browser in the other and I've been experiencing this same issue: lots of black frames and also a huge delay on the video. This doesn't happen if I try to use a Nexus 5 instead of Glass, so was wondering if you have any idea if the same fix that is in master for this issue might help my use case with Glass.
I'm already trying to compile the latest version to try it out. So another thing I wanted to ask(maybe I shouldn't ask in this issue :smile: ) is, if I want to measure where is the time being spent (encoding, decoding, traveling through the network) where do you recommend to do it? (I might have to do this if after compiling master the problem still persists)
Any help is welcome, thanks!
@nbermudezs that's a fun use-case! We did something similar some time ago (before releasing OpenWebRTC publicly): http://www.ericsson.com/research-blog/context-aware-communication/field-service-support-google-glass-webrtc/
I seem to recall that we had to use the hardware accelerated video coding, provided by OpenMax, to get video to flow nicely. @Rugvip will know more as it was his Thesis project :smiley:
Sounds like you should create a new Issue for this, though.
Thanks for the reply @stefanalund. That blog post was the main reason I'm trying to use OpenWebRTC in G. Glass. I'll open a new issue and hope @Rugvip can provide some insights about this.
Hi, my team is also working on OpenWebRTC for mobile app and am having this black frame dropping issue. Is there a new update on this? Quite important for us!
Hi guys, I recently work on Android solutions, and also find this problem, do you have any ideas that I can do for this ? to improve the black screen problem, thanks a lot.
