toxcore
toxcore copied to clipboard
irungentoo
Don't drop video keyframes.
Currently as I understand it toxcore shoves all the video data over UDP and hopes it reaches the destination.
This creates horrible video corruption when a key frame is lost. On desktop streaming, intra frames are super rare so we can easily go 10s at a time with completely garbled video output. It's a common thing when calling EchoBot for example, with both uTox and qTox.
Toxcore should try on the sending end to retransmit iframes until it either gets an ACK or the next iframe is ready, and on the receiving end it shouldn't give Tox clients inter frames that depend on missing intra frames.
Maybe we can have "backup" keyframes in case they get dropped? Just trying to think outside the box for alternatives. Refusing to drop keyframes is probably the best option.
I think the standard solution for this problem is buffering. If you drop any frame (be it a keyframe or an inter frame), the image will look like garbage. The receiver should instead wait (e.g. 1 second), while the sender should resend any frame that did not arrive. If a frame needs to be dropped, than all inter frames between the last keyframe and the next need to be dropped as well. The viewer would then only experience a lower frame rate, instead of seeing noise. If I am not mistaken, Jitsi works this way.
Easier said than done, though.
Dropping interframes isn't nearly as bad, because dropping a keyframe means every frame between that and the next keyframe will be corrupted (by my undersatnding).
GrayHatter claims this is a client-side issue. Is this actually true? Does the toxcore api have a mechanism for sending lossless packets? Are there even use cases where compressed video would be sent using lossy packets for keyframes?
Let me know if there's something I'm misunderstanding. I really want to tackle this issue. Performance of video chat is really important wrt competing with hangouts and the like. Really, really important.
As far as I can tell, toxcore sends lossy packets and has no configuration for doing otherwise:
https://github.com/irungentoo/toxcore/blob/master/toxav/rtp.c#L146