Recorded videos are partially broken

[hardcore-sushi]

trafficstars

Here's how the video recording is implemented in DroidFS:

Frames are recorded with the CameraX library, which also encodes video and audio with MediaCodec. In a conventional implementation, this would then use MediaMuxer to mux audio & video to a file. However, in DroidFS, we want the output to be encrypted on-the-fly, before anything is written to disk. Therefore, some components of CameraX have been modified to use FFmpeg as a muxer, thereby allowing the output to be encrypted on-the-fly. These modified files can be browsed here.

The problem is that sometimes the video freezes during the recording and although the final video file can be played, it seems to contain errors. When played with ExoPlayer, you can see things like this in the logcat:

3750 E MediaCodecAudioRenderer: Audio sink error
3750 E MediaCodecAudioRenderer:   com.google.android.exoplayer2.audio.AudioSink$UnexpectedDiscontinuityException: Unexpected audio track timestamp discontinuity: expected 1000000150333, got 1000005122291
3750 E MediaCodecAudioRenderer:       at com.google.android.exoplayer2.audio.DefaultAudioSink.handleBuffer(DefaultAudioSink.java:767)
3750 E MediaCodecAudioRenderer:       at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.processOutputBuffer(MediaCodecAudioRenderer.java:41)
3750 E MediaCodecAudioRenderer:       at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:362)
3750 E MediaCodecAudioRenderer:       at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:72)
3750 E MediaCodecAudioRenderer:       at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:150)
3750 E MediaCodecAudioRenderer:       at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:225)
3750 E MediaCodecAudioRenderer:       at android.os.Handler.dispatchMessage(Handler.java:102)
3750 E MediaCodecAudioRenderer:       at android.os.Looper.loop(Looper.java:223)
3750 E MediaCodecAudioRenderer:       at android.os.HandlerThread.run(HandlerThread.java:67)

Here is the output of a simple ffprobe:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55cd37758980] st: 0 edit list: 1 Missing key frame while searching for timestamp: 0
[h264 @ 0x55cd37759f00] missing picture in access unit with size 30
[h264 @ 0x55cd37759f00] no frame!
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'v.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.4.101
  Duration: 00:00:03.00, start: 0.000000, bitrate: 21221 kb/s
  Stream #0:0[0x1](und): Video: h264 (Baseline) (avc1 / 0x31637661), yuvj420p(pc, bt470bg/bt470bg/smpte170m, progressive), 1920x1080, 21218 kb/s, 29.04 fps, 30 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
    Side data:
      displaymatrix: rotation of -90.00 degrees
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 156 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]

I think the problem is likely to come from my muxing implementation with FFmpeg or from the changes I made to CameraX to work with FFmpeg.

I modified the Recorder class from CameraX to use a MediaMuxer interface instead of the Android MediaMuxer which doesn't allow on-the-fly encryption. CameraActivity can then configure it to use FFmpegMuxer, which is a kind of JNI wrapper around libmux, which is using FFmpeg to mux video & audio. Note that I also modified EncoderImpl from CameraX so it doesn't drop codec config data. You can see the changes made to CameraX with diff. For example with the Recorder class:

diff -u app/src/main/java/androidx/camera/video/SucklessRecorder.java app/src/main/java/androidx/camera/video/originals/base/Recorder.java

I found only one example of a FFmpeg muxing of videos encoded with MediaCodec. However, the code is pretty old and the resulting videos also seem broken. My knowledge of video formats/encoding/muxing and FFmpeg is very limited. That's why I'm looking for help with this issue. Thank you very much to all those who would have an idea to solve this problem.

[CodeCubeNeo]

Is there any way to recover the corrupted video? I have exported it, and it does play the sound, the time is shown, but there is no video.

[hardcore-sushi]

Does this happen with every video?

[CodeCubeNeo]

Yes. The built-in player cant play it, but when the file is exported, the duration is correct and the sound plays, but as said without the video

[hardcore-sushi]

OK, what video player do you use? Can you try with VLC or MPV?

[CodeCubeNeo]

On both is the same result - no video

[hardcore-sushi]

OK. What's the result of ffprobe on the file? Can you attach a logcat of when the video is recorded and when it's played with the built-in video player please?

Ensure you have the latest DroidFS version (v2.0.1) and that the app has permission to record videos. You can also try revoking audio recording permission to DroidFS.

[CodeCubeNeo]

This is the ffprobe output: [h264 @ 0x7f69857400] missing picture in access unit with size 108517 [extract_extradata @ 0x7f698b1230] No start code is found. ./corrupt_video_test.mp4: Invalid data found when processing input

For this file

https://user-images.githubusercontent.com/61159092/236743314-dc8065c1-7d19-4b2d-9bf4-1e54ba3e31da.mp4

How/where do i get the logcat?

[CodeCubeNeo]

Sorry for asking before researching myself. I hope this logcat is useful logcat.txt

[hardcore-sushi]

I can't find the cause of the problem. What's your Android version? Are you using a custom ROM? Can you please revoke audio recording permission to DroidFS then retry to record a video?

[CodeCubeNeo]

I am using stock android 9 on Huawei Mate 9. If I revoke the microphone permission, the built in player can't play the video and the exported version is now the same as before, but without the sound. Thus the ffprobe output is the same. Was the logcat useful? and how to check if video info is in the file (the previously uploaded clip)?

[hardcore-sushi]

I didn't find any relevant errors in your logcat. I think video data are in the file because it's 5MB large.

b3a25e03e7de8de73a6ba2491fef0338c588210c should provide us more information as it enables logging of libav related errors.

[CodeCubeNeo]

Can you maybe publish a dev build somewhere so that i can give the now more precise logcat (if i understood clrrectly)?

[hardcore-sushi]

The new version has been published. You can try with it.