ladybird icon indicating copy to clipboard operation
ladybird copied to clipboard

LibMedia+LibWeb: Demux videos with FFmpeg and play audio with videos

Open Lubrsi opened this issue 1 year ago • 1 comments

This acts as a drop-in replacement for our MatroskaDemuxer, giving us access to containers other than Matroska, with the biggest one being MP4.

Alongside this, since we already demux the audio with FFmpeg and create audio tracks regardless of media type, we can just play the audio tracks with the video element! However, audio/video sync is not perfect, but that's not the goal of this PR :^)

For example, with MediaSource disabled (i.e. removing Window from the Exposed attribute on MediaSource and ManagedMediaSource), we can watch 360p h.264 mp4 YouTube videos!

https://github.com/user-attachments/assets/00cafce6-34fe-428b-9476-77ca33b21812

Note that there some improvements we could make here, e.g. dropping the MatroskaDemuxer and passing FFmpeg packets around directly, but that can be done in the future.

There's also some work we've got to do with keeping the data alive whilst the FFmpeg context exists. GCing a media element is very likely to cause a UAF with the audio stream at the moment, because audio decoding and playing is ran in a different thread, and GCing the media element destroys the underlying data buffer without coordinating with the other thread.

Lubrsi avatar Mar 10 '25 16:03 Lubrsi

cc @Zaggy1024

ADKaster avatar Mar 10 '25 16:03 ADKaster