How to play standalone h264 video and pcm audio bytes stream synchronously

[yikuo123]

I receive standalone h264 bytes stream and pcm bytes stream separately, how can I play them together synchronously with MediaElement or MediaPlayerElement? Thanks a lot!

[brabebhin]

Hi

See https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/play-audio-and-video-with-mediaplayer#use-mediatimelinecontroller-to-synchronize-content-across-multiple-players

[lukasf]

I always had the idea to allow adding external audio streams. But until that is actually implemented, the only way is to use MediaTimelineController, as @brabebhin already pointed out.

[yikuo123]

Thanks! I'll have a try.

I'm looking forward to adding external audio stream, it is what I really want.

[lukasf]

I did not say that I was going to add this anytime soon though. There are some difficulties with that.

Also, if you are receiving these as live streams, then it might be difficult to get them 100% sync, because each stream will have different latency. Both streams would start playback at the current position where they started receiving, with might be slightly different for video and audio stream. I am not sure if it is possible to sync them up correctly in a reliable way, even if we had support for this in our lib. That is why corresponding streams are usually not sent as raw streams, but muxed in a container stream. Or e.g. for DASH format, it has separate streams for audio and video, but each is packed into a MP4 container stream, and that MP4 stream contains the synced up PTS timestamps.

If you work with files, things are much easier. Or if you can make sure that both streams start at the exact same position, then syncing up should be possible.