Reducing sync delay by throwing away samples

[stewartsiu]

I started using SWYH-RS again recently after some compatibility issues with my streamer (long story that I might start another thread about), and while I got it to a usable state, I'm still having some issues with audio-video sync when I watch youtube videos. Naturally the audio is slightly delayed, but sometimes it's negligible (<50ms), and sometimes it's much more (>200ms), and that's randomly determined when I first start the program and stays that way afterwards. This gives me an idea:

How about we add a config that allows us to throw away x milliseconds of samples at the beginning of the program? I know that when the sync is bad, after I hit pause the sound keeps on playing for a bit. If we could throw away some samples when writing to swyh.wav I'd think that could help with the sync.

[dheijl]

I think your best bet is to use VLC to watch youtube, because VLC allows you to specify an offset for the audio in relation to the video. Normally swyh-rs audio will be trailing behind the video, so throwing away audio samples would only make things worse (I think).

[stewartsiu]

VLC: I'd love to use VLC to watch youtube, but Google makes it very difficult, as they wouldn't want to lose ads. Trailing behind: Yes audio would be trailing behind. Say the video counts 1 to 10 quickly. When the video is at 5, the audio might be at 4. But if we throw away the audio sample for 5, when the video is at 6, the audio would skip from 4 to 6 directly, thus catching up with the video. Makes sense?

[dheijl]

If I "throw away" samples, with what should I replace them? The audio stream is embedded in the video stream, the only option would be to delay the video stream. So I'm afraid your idea wouldn't work...

[stewartsiu]

Pseudocode of what I mean if we throw away k samples:

sample_removed = False if size(fifo_buffer) >=k and not sample_removed { fifo_buffer = fifo_buffer[k..] sample_removed = True }

This is after you’ve read the audio stream so no video stream is involved here. In a perfect world where the PC and the renderer have no hiccups I assume it wouldn’t change the audio video sync and would only cause a brief silence when the fifo_buffer is emptied.

On Saturday, February 24, 2024, dheijl @.***> wrote:

If I "throw away" samples, with what should I replace them? The audio stream is embedded in the video stream, the only option would be to delay the video stream. So I'm afraid your idea wouldn't work...

— Reply to this email directly, view it on GitHub https://github.com/dheijl/swyh-rs/issues/119#issuecomment-1962757472, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACT4UNHNOABZYTYB3234TCLYVJ2HXAVCNFSM6AAAAABDYIAVW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRSG42TONBXGI . You are receiving this because you authored the thread.Message ID: @.***>

[stewartsiu]

To paraphrase, i’m not suggesting a general way to change the time distance between video and audio - instead I’m proposing a way to catch up when too big of an audio buffer has been built up due to some hiccups. Ideally it should be done on the swyh.wav, but maybe it’d work with the buffer in memory too.

[dheijl]

The delay is not built up in memory or in swyh--rs, the delay is built up in the network transmission and in the receiving device, and you have no control over those. Throwing away bytes will do exactly nothing except cause silence periods. The audio will not arrive faster because you throw some of it away. It arrives in sync with video. You can not speed it up.

[stewartsiu]

Yes, throwing away bytes will only cause silence periods if the delay due to network and receiving device is constant.

My hypothesis on my device is that there is sometimes a spike in latency at program startup that subsequently goes down. Let's say your normal latency is 50ms, but somehow there is a latency spike of 200ms. After the spike, even though the delay due to network and receiving device goes back to 50ms, the audio would still be behind by 200ms, with 150ms of audio built up in a buffer somewhere, so you can improve the sync by removing ~150ms of sample. My hypothesis came partly from the observation that when I have a 200ms+ delay, and the sync would improve if I stop and restart swyh-rs, but I don't want to do that because it causes other problems.

[dheijl]

You can never improve the sync by throwing away samples because the samples are provided by the OS at a fixed rate and you can not read them any faster than that rate (that is in sync with the video).

[stewartsiu]

... because the samples are provided by the OS at a fixed rate and you can not read them any faster than that rate (that is in sync with the video).

Hm, I think my hypothesis wouldn't contradict what you wrote above. Let's say we have a perfect setup with no sync latency, now the renderer (that doesn't know anything about video) suddenly stops reading the stream / playing the sound for 200ms, and resumes again from where it stopped. 200ms sync latency would be added by now, and the renderer would be playing each sample 200 ms after the OS provided them. Wouldn't such a situation benefit from some samples being skipped?

I understand that my video sync issue might not be relevant for most swyh users, so I might just try it myself to see what happens. Mr. Heijl, if you could suggest a few places in the source code to implement the sample skipping, I can give it a try in my spare time and report back.

[dheijl]

I suppose the best place to drop samples would be at the source as they arrive :

file: src\utils\audiodevices.rs

function: fn wave_reader<T>(..).

[dheijl]

I'll close this now, but feel free to reopen if you have new information.

[stewartsiu]

My finding is that throwing away samples at various times or inserting samples jiggle the latency around, but the effect seems mostly random, and throwing away too many samples will trigger stuttering. I tried the sample skipping at four different places in the code, one at audiodevices.rs as recommended, one at write() in rwstream.rs, two at read() in rwstream.rs, and the effect is more or less the same. I also played with the interactions with sending_silence. It's easy to throw the latency off balance, but it's very hard to predict what the outcome would be, so it's as if there is some kind of compensation happens when samples are skipped.

[dheijl]

Thanks for testing this! But it confirms what I thought: throwing away samples is not a viable solution for reducing latency.