a/v sync issue with fragmentation mode ?

[rhwu]

First of all, I'm not sure if this is a muxer issue or the way how I used it was wrong.

I used minimp4 for a live stream recording case. Live video and audio were captured, encoded and fed to the muxer. The duration parameter to the MP4E_put_sample function is calculated from the audio/video capture timestamp, e.g. TSn - TSn-1.

The recorded mp4 file played fine at the beginning, but you can tell the gap between the audio and the video is accumulating. The out of sync becomes noticeable after around 1 minute of play and gets worse as time passes.

So I have two questions:

1. What's the right way to compute audio and video sample duration for muxer in my case (video framerate may not be fixed)?

2. Looking into the code, it seems like the audio duration is set in tfhd as the default sample duration, while the video duration is set in trun as sample duration. What are the difference? Am I suppose to use a fix value for the audio duration, because it's the "default sample duration"?

Thanks!

[lieff]

Sorry for the delay, I miss notification for some reason.

In mp4f mode sample duration should be passed directly from MP4E_put_sample. Problem most likely in fps or sample rate. I tried stream.h264 and stream.pcm from minirtmp with this patch:

@@ -3,18 +3,18 @@
 #include <sys/types.h>
 #include <stddef.h>
 typedef size_t ssize_t;
 #endif
 #include "minimp4.h"
-#define ENABLE_AUDIO 0
+#define ENABLE_AUDIO 1
 #if ENABLE_AUDIO
 #include <fdk-aac/aacenc_lib.h>
 #include <fdk-aac/aacdecoder_lib.h>
 #define AUDIO_RATE 12000
 #endif
 
-#define VIDEO_FPS 30
+#define VIDEO_FPS 24
 
 static uint8_t *preload(const char *path, ssize_t *data_size)
 {
     FILE *file = fopen(path, "rb");
     uint8_t *data;
@@ -186,11 +186,11 @@ int main(int argc, char **argv)
     int is_hevc = (0 != strstr(argv[1], "265")) || (0 != strstr(argv[i], "hevc"));
 
     MP4E_mux_t *mux;
     mp4_h26x_writer_t mp4wr;
     mux = MP4E_open(sequential_mode, fragmentation_mode, fout, write_callback);
-    if (MP4E_STATUS_OK != mp4_h26x_write_init(&mp4wr, mux, 352, 288, is_hevc))
+    if (MP4E_STATUS_OK != mp4_h26x_write_init(&mp4wr, mux, 240, 160, is_hevc))
     {
         printf("error: mp4_h26x_write_init failed\n");
         exit(1);
     }

Mux:

minimp4_x86 -m -f stream.h264 out.mp4 

And produced file shows normal sync till end (note 30->24 fps change).