javacv
javacv copied to clipboard
bytedeco
Question: Best Way Synchronize Video and Audio When Playing Videos using FFMPEG
Problem: When reading in some videos, audio and images are returned by FFmpegFrameGrabber is in X second chunks. For example, in one mp4 I'll get a sequence of images in a row when calling grab() that's one seconds worth. After the images I'll then get 1 second's worth of audio. This is problematic because I want to play the video in real-time with audio and video properly synchronized. I've yet to think of a good solution.
Possible Solutions:
-
The Ideal Solution Ideally I would get one image then be feed all the audio for that image in an interleaved fashion. Is that possible by changing some setting I'm not aware of?
-
Cache Images and Audio I can cache images and audio and synchronize it when playing it back to the user. I'm estimating this to require more than 1 gig of memory and cause a noticeable pause after seeking to a new frame. So this is undesirable.
-
Abuse Seeking Could grab an image then read the video until I found the audio I need then seek to the next image frame. This is likely to be very slow.
-
Launch two instance of FFmpegFrameGrabber. One would read audio and the other images. This might be the best solution. Less memory and cpu intensity than other solutions. I'm not sure if some codecs use hardware acceleration. That could be problematic if all the hardware resources are used by one instance.
Thanks for the help and this great library!
Do they buffer the compressed images? Is it possible to read in compressed images into memory then decompress it later on? While almost all modern machines could handle that amount of memory it's still a lot. Here's how I came up with that number for a video from my Galaxy S8. 400030003 (image) * 30 (frames) / 1024 / 1024 = 1030 MB.
I implemented the two instances of FFmpegFrameGrabbe approach. That's working well but I should profile to see if there's a performance hit from reading the video twice. I'm not sure if grabSamples() skips image data or decodes it. Might end up needing to buffer no matter what. Hard drives with a spinning disk can experience very long delays reading.
@anotherche Would you have any ideas about the best way to go about this?
@jpsacha Has also enhanced the JavaFxPlayVideoAndAudio sample since then: https://github.com/bytedeco/javacv/blob/master/samples/JavaFxPlayVideoAndAudio.java @lessthanoptimal Could you give that a try and see if it works well enough for your needs?
@saudet Glad to see these issues being revived! The application I was working on at the time is basically dead at this point. There might be another video editing project in the future. If that happens then I can test all of this out.