saki4510t
TimeLapseRecording with MediaMuxer
Hi. I saw your project "AudioVideoRecordingSample". You record audio and video using MediaCodec and MediaMuxer. I want to use this project, and add "TimeLapse" feature. How to add TimeLapse to MediaMuxer? I saw your second project "TimeLapseRecordingSample", but you didn't use MediaMuxer for creating file.
Hello,
There are several ways to make time-lapse movie. In TimeLapseREcordingSample,
- Get images from camera
- Encode images using MediaCodec
- Save encoded images as file with original format Audio and video data store into each file.
- After stop recording, read encoded images from own format file and write them as .mp4 file using MediaMuxer.
And you can see the step4 in TLMediaMovieBuilder.java.
with best regards, saki
Thank you very much for your answer. Could you please tell me in which steps we have to do the "speed up" thing?
And what do we have to do? Only change the presentationTimeUs? Anything else?
Hello,
The target of this project is manually made intermittent video images into one movie file(.mp4). And it will be little bit different from usual time-lapse movie. Resulted movie frame rate is same as usual video recording.
If you want to speed up(I assume "speed up" means you want to play resulted video on fast-forward), you need to change presentationTimeUs to smaller value when passing encoded frame to MediaMuxer on step4. Don't forget presentationTimeUs should be monotonic and has offset. You will need to subtract start time from presentationTimeUs, multiply speed ratio(smaller than 1 for speed up, greater than 1 for slow down), add start time then pass them to MediaMuxer.
saki
Thank you very much! It works. But there is another problem. Since I speed up video ( x3), the framerate of video increase. For example: Before speeding up the framerate was 30fps. Now it is 90fps. And the default video player of phone cannot play the video normally. If I copy the video to computer, the video player of computer works well. Is it an error of encoding process?
Hello,
Well I don't confirm the issue but I assume it is just because your device does not have enough ability of decoding and/or ability of rendering(most smartphones/tablets only support up-to 60fps rendering and some of them only support up-to 30fps) and/or ability of loading data from storage for such high speed frame rate.
If you make your own player, I assume you need to check number of input frames to MediaCodec decoder and output(decoded) frames from MediaCodec decoder and the difference of them become specific value, it will be better to drop some input frames.
Or if you want to playback correctly with default player on Android, I assume it is better to drop some frames when encoding(don't drop key frames though).
saki
Yes, I'm finding the way to drop frames when encoding. But I don't know how to define which frames is the key frames. I know that in AVC stream there are I, B, P frames. They may be dependent upon other frames, so I can't drop them arbitrarily. But know to define the key frames with MediaCodec?
Hi,
If a frame data encoded by MediaCodec is key frame, they have BUFFER_FLAG_KEY_FRAME flag(or BUFFER_FLAG_SYNC_FRAME for build target <API21).
FYI, as I know, Android officially only support baseline profile for H.264 at least for encoding now and baseline profile has no B-frame. So in most case now, BUFFER_FLAG_KEY_FRAME means the frame is I-Frame and otherwise the frame is P-frame when encoding. This may change on future release of Android though.
saki
I tried to keep all the I-frames. But the result is not good. The wiki says: A P‑frame ('Predicted picture') holds only the changes in the image from the previous frame. That means we cannot drop the P frames? I assume we have a sequence: I2 P1 P2 P3 P4 I2. What happens if I drop P2? Because P3 uses data from P2.
It's depends on what you want to do. Dropping middle P-frame reduce resulted movie. But if your video scene has only little changes between each frame, it will be able to ignore and most devices still can playback the movie. If your video scene has lot of difference between each frame(like a sport scene), drooping middle P-frame will impact strongly on movie quality as you can see when bitrate is not enough and in this case you need to use shorter I-frame intervals.
Well I'm not sure what actually you want to do,
if you just want to make faster speeded movie, as I mentioned before, it is better to drop some frames when encoding(not after encoding, for example, if you just want to make x3 speed movie and target playback fps is 30fps, only 10fps is enough for input).
If you don't know what speed you want to playback or you want to decide playback speed when decoding and want to use default player, I assume you need use shorter enough I-frame intervals and drop continuous P-frames before I-frame. Or omit all P-frame(all frames are I-frame / use still images instead).
I want to make x3 speed movie. Here is what I found after a lot of tests: If we set KEY_IFRAME_INTERVAL = a and KEY_FRAME_RATE = b, there will be axb P- frames between every 2 I-frames. For example: if KEY_IFRAME_INTERVAL = 2, KEY_FRAME_RATE = 30, there will be 60 P-frames between 2 I-frames.
Here is problem with dropping P-frames. I assume that we have 30 P-frames between 2 I-frames: I1 P1 P2 P3 .... P30 I2. If I keep P1, P2,.. P10 and drop P11, P12, .. P30 => There is no problem with display colors, but resulted video is not continuous. If I keep P1, P4, P7,...., P28 => There is problem with display colors.
I tried to set FPS = 10 (and didn't drop any P-frames) as you suggested. But it doesn't work. Because FPS = 10 doesn't mean that every second encoder generates 10 frames. FPS = 10 or 30 doesn't affect to amount of generated P-frames every second.
@sunshinetpu were you able to speed up movie by 3x? I am also stuck here with the same problem as in your previous comment. If i drop frames, resulted video is not good in terms of continuity and colors. And if i set FPS to 10 and didn't drop any frame then it has no effect.
You should drop frames before they go into encoder. If you can set your camera to 10 FPS and change presentationTimeUs to match 30 FPS, your result may be smoother. If your player supports 60 FPS (all modern mobile devices do, but this depends also on network quality), you can try to set the camera to 15 FPS and tune presentationTimeUs to match 60 FPS.
I dropped frames before sending them to encoder and was able to record video in 7fps without any frame distortion. But when i tried dividing presentationTimeUs by 3x for timelapse, i got unexpected result. While recording fps was 7 and actual video time was 10 sec and after 3x timelapse, output video has 21 fps and time was 3 sec, as expected. While playing video, it is moving back and forward many times, see attachment, specifically video progress bar. This only happens when i play video on android device but on my computer it is playing fine.
I am not able to figure out what the problem is.
This could be a limitation of the system video player on your Android device: it could have problems with playback at non-standard 21 FPS. OTOH, you should run analysis of the movie with ffprobe -show_frames.
If i record only video, then it is playing fine as expected. But if i record audio too, problem occurs and audio is not speeding up or slowing down even after changing the presentationTimeUs.
@Zer0bee I am not sure I can understand how you expect audio recording to behave for TimeLapse. Do you expect the sound at higher pitch, or sampled every now and then, or what?
@alexcohn My goal was to time lapse both video and audio but then i realise that for audio i have to implement time stretching algorithm, simply changing presentationTimeUs will not time lapse audio.
I'm still trying to figure out the way to time lapse audio while recording.
Hi @Zer0bee did you find an answer to the problem of audio timelapse?
