Pre-Event Motion Capture Buffer

[username7291]

I test now a long time this program, and found out that the video artifacts problem always occur at the beginning of the Video. It looks like a broken internet video that has missing bytes. I tryed buffers from 1001-1064, i read it maybe requires a iframe(i assume if the none it begins on the first new it gets), the default is all 60 frames. First i want to fix this by adjusting INTRAPERIOD it but then i seen its already in the source. ip = ((long long)cfg_val[c_video_buffer]*(long long)cfg_val[c_video_fps]) / 4000; means in my case about every 6th frame. In a 1001 ms Buffer are now about 3 I-Frames. Only 1 is required, so the video should be put valid together. But why the Video is broken anyway? Its seem you on the fly change this value to all 60 frames after a Motion is Detected. Is it possible that mmal/mp4box/h264 standard dont like that?

[username7291]

I found out the buffer handling is broken, everytime a video is broken it writes the whole buffer to the File. fileSizeCirularBuffer, copyfromstart,copyfromend, cb_wrap {2021/02/11 16:29:43} prebuffer 840028 236088 603912 1 rpos: 5478 wpos:5482 wptr:790043 buffer left after wprtr: 49957 The last iframe dont match into the buffer it should be write a part at the end and the rest on the start of the buffer. I assume dumps to the file 5478-609390, 0-236088 thats wrong. Cause there are about 18 valid frames(means too a valid iframe at the beginning) in the Video and after it comes frames, then all valid again. It should be about 5478-840028 and 0-5478 if on rpos is a valid iframe. If not first from the first valid iframe. I think here is not much activity anymore and i have to fix it myself? The code is not well dokumented.. Maybe its easyer to rewrite the buffer to a non cicular version. Write to the buffer[0] if a iframe arrives and the next time in buffer[0] if the next iframe arrives. How long the buffer should, can set via Intra Period(in limits but better as a non working pre-event buffer)

[roberttidey]

I have just starting checking this out. It does look like the first few frames can be repeated but after that the pre trigger buffer works as intended to include the video leading up to the trigger

The buffering code was originally imported from another video app a long time ago so it will need some investigation to work out what is wrong there. The basic principle is that when a real video recording starts the that data is written to the output file. When the recording stops then the buffer is searched to find the earliest i-frame and that part of the buffer is then pre-pended to the output file.

[username7291]

i done for the first try: if (copy_from_start + copy_from_end>=cb_len) {printLog("fix\n"); copy_from_start=0; copy_from_end=cb_wptr-iframe_buff[iframe_buff_rpos];} if (copy_from_start) fwrite(cb_buff, 1, copy_from_start, h264output_file);

That writes the first part(about 18 first frames are always valid) and skip the second part, like old invalid frames that dont belong after the first part and make the video corruption. But thats untested. If it trys to write the full buffer size it must be wrong. The chance that IFrame-otherFrames-IFrame is exactly the size of the Buffer is very very low. I found on the net a guy, he told about that bug 2017 already.

[roberttidey]

Thanks a lot for your work on this.

Can you let me have the full surrounding code you are changing so I can try it out? Or put in a pull request if you have forked the code.

[username7291]

thats RaspiMCam.c

    if (copy_from_start + copy_from_end>=cb_len) {printLog("fix\n"); copy_from_start=0; copy_from_end=cb_wptr-iframe_buff[iframe_buff_rpos];}

before long fileSizeCircularBuffer = copy_from_start + copy_from_end + header_wptr; then if (copy_from_start + copy_from_end>=cb_len) {printLog("fix2\n"); copy_from_start=0; copy_from_end=cb_wptr-iframe_buff[iframe_buff_rpos];} before fseek(h264output_file, 0, SEEK_SET); fwrite(header_bytes, 1, header_wptr, h264output_file); fwrite(cb_buff + iframe_buff[iframe_buff_rpos], 1, copy_from_end, h264output_file); if (copy_from_start) fwrite(cb_buff, 1, copy_from_start, h264output_file);

of the last line the zero check of copy_from_start is new. Today i had 8 Videos where the fix is applied, all of them are valid without errors.

[roberttidey]

OK. I'll give that a trial and then merge as appropriate. Thanks.

[roberttidey]

I have tried that but I don't see it going into those fix statements after trying multiple recordings.

For reference in video_start I have

         int copy_from_end, copy_from_start;
        copy_from_end = cb_len - iframe_buff[iframe_buff_rpos];
        copy_from_start = cb_len - copy_from_end;
        copy_from_start = cb_wptr < copy_from_start ? cb_wptr : copy_from_start;
        if(!cb_wrap) {
           copy_from_start = cb_wptr;
           copy_from_end = 0;
        }
		if(copy_from_start + copy_from_end >= cb_len) {
			printLog("buffer fix start\n");
			copy_from_start = 0;
			copy_from_end = cb_wptr - iframe_buff[iframe_buff_rpos];
		}
		long fileSizeCircularBuffer = copy_from_start + copy_from_end + header_wptr;
        fseek(h264output_file, fileSizeCircularBuffer, SEEK_SET);

and in video_stop

      int copy_from_end, copy_from_start;
        copy_from_end = cb_len - iframe_buff[iframe_buff_rpos];
        copy_from_start = cb_len - copy_from_end;
        copy_from_start = cb_wptr < copy_from_start ? cb_wptr : copy_from_start;
        if(!cb_wrap) {
          copy_from_start = cb_wptr;
          copy_from_end = 0;
        }
		if(copy_from_start + copy_from_end >= cb_len) {
			printLog("buffer fix stop\n");
			copy_from_start = 0;
			copy_from_end = cb_wptr - iframe_buff[iframe_buff_rpos];
		}
        fseek(h264output_file, 0, SEEK_SET);
        fwrite(header_bytes, 1, header_wptr, h264output_file);
        fwrite(cb_buff + iframe_buff[iframe_buff_rpos], 1, copy_from_end, h264output_file);
        fwrite(cb_buff, 1, copy_from_start, h264output_file);

So either I have got the changes wrong or there is a difference in your set up e.g. buffer size that is making a difference.

[username7291]

Looks right but copy_from_start zero check in front of the write is missing, dont know if it is allowed to pass 0 to fwrite(). Do you have broken Videos with enabled prebuffer? If not you cant see the difference. Whats your buffer size? I tryed 800, 1000, 1001,1004,1008,1016, 1032, 1064, 1092, 1500. 1500 is already to much, i play the videos want see what cause the detection and skip to the next. If have to wait before something moves this takes to long. If you have a really large buffer it should happend fewer. Video is like abcDEFGH in the buffer. On this Bug the program takes D(Iframe)-H and abc. The write is on G. Means H is an old frame, and abc are unusable frames. DEFGHabcIJKLM...., the new letters are the unbuffered Video that follows later. Cause the video gets broken. The fix takes the D(selected Iframe) to G the last frame that belongs to the Iframe. With a very large buffer the write position is less often near the End of the Buffer and trigger that bug less often. But it happens the guy who reported that 2017 used a 5000ms Buffer. Today i get again 8 Times the fix(800ms Buffer) about 30-40%,, all Videos are errorfree.

[roberttidey]

I did try various buffer sizes including 500 but I'll try more intensively with a lower value to see if I can get the fix to activate.

I previously had not had any difficulty playing buffered videos.

What video resolution / frame rate are you using? I typically use the full view 972p setting but I did try testing with HD 1080p.

[username7291]

1920x1080, 22fps, Bitrate 8400000(higher fps cause missing frames with Motion Detection enabled). With the old camera modules you use one of the binning modes cause of light problems... But that means too the frame size is lower and a hit chance too. HQ Camera can do binning of 1920x1080. Maybe check if the Bitrate of the Video(MediaInfo) matching the settings, omx gives sometimes much lower results on lowres.

[roberttidey]

I don't use the HQ camera but I'll do some more testing with different values.

In the meantime I added the modifications anyway to the github.

[username7291]

Works still very well, but in 10 days i had 2 broken Videos, its nothing against 8 per day but if its possible to fix that too.. One hits the fix but isnt fixed, one dont hit the fix. I seen DEBUG 3 in the Log, that means nothing happends on every Video end. Then there sometimes DEBUG 2 in the log, but the output says not much, i added some more infos: [code] for(i = iframe_buff_rpos; i != iframe_buff_wpos; i = (i + 1) % IFRAME_BUFSIZE) { int p = iframe_buff[i]; if(cb_buff[p] != 0 || cb_buff[p+1] != 0 || cb_buff[p+2] != 0 || cb_buff[p+3] != 1) {
printLog("DEBUG 2: %i, %i %d %d %d\n", iframe_buff_rpos, iframe_buff_wpos, cb_wptr, cb_len,p);
iframe_buff_rpos = (iframe_buff_rpos + 1) % IFRAME_BUFSIZE; error("Error in iframe list", 0); } } [/code] And thats what i get: rpos wpos cb_wptr cb_len p {2021/02/24 11:44:31} DEBUG 2: 2292, 2296 125919 860000 859999 {2021/02/24 11:44:31} DEBUG 2: 2293, 2296 170260 860000 859999 {2021/02/24 11:44:31} DEBUG 2: 2294, 2296 190224 860000 859999 {2021/02/24 11:44:31} DEBUG 2: 2295, 2296 201547 860000 859999 It tryes to reads beyond the buffer. I assume gcc breaks already at [p] !=0, cause linux dont kill the program(illegal memory access). So far its clear a i-frame cant be fit at the end of the Buffer, the identifier is already 4 Bytes. There is a type of adressing made of IFRAME_BUFSIZE, its 131072 Bytes. But i dont know how its possible that a IFRAME is always small like that(doesnt matter the Bitrate or Resolution). If i save a jpeg from 1920x1080 its already 176.498 Bytes thats lower as an 80% compressed jpeg.

[username7291]

Okay i understand this Buffer stuff a bit better. Its just 131072 Pointer Array that goes to zero if its on last Element. The Debug 2 Routine seems just a valid iframe check, that skips to next iframe on an error(no identifier match). If 1 Byte is at the End and 3 Bytes are at the start its not an error, only the routine is not able to handle a buffer wrap. I done this: if (p+3<cb_len) { if(cb_buff[p] != 0 || cb_buff[p+1] != 0 || cb_buff[p+2] != 0 || cb_buff[p+3] != 1) {
printLog("DEBUG 2: %i, %i %d %d %d\n", iframe_buff_rpos, iframe_buff_wpos, cb_wptr, cb_len,p); iframe_buff_rpos = (iframe_buff_rpos + 1) % IFRAME_BUFSIZE; error("Error in iframe list", 0); } } //valid area check Cause wrap its no real error, they is no reason for skipping I-Frames(If the rest of the Buffer handling works at least). Cleaner would be to support check over the wrap. I dont know if this solves this 2 Video errors in 10 Days, takes long to check. If not it should keep the Log clean at least.

[username7291]

The Intraframe setting is pretty buggy, if you set 6 you get K-Frames at 0,12,24,36. Expected by the description would be 0,6,12,18. Cause the MCam request very often a new I-Frames if there is no keyframe in 1/4 of the Buffer. The other thing is mmal dont care much of the requested Bitrate, it can match the request, later half of the request, later twice the requested. Cause i assume the overwrite check doesnt work, Frames may overwrite more than one Frame cause there are much bigger than expected by Bitrate Settings. Cause i doubled the Buffersize, request iframe only if none arrived 1/3 of Buffer use only last 4 iframes of the buffer. It looks better again, there is now 1 Errortype left. If it occurs, its always at the 25.Frame, and only 1 Frame contain a error. Its not a Keyframe(0,12,24,36,50,170 are), its withhin the Buffer. My guess something blocks too long at MotionDetection(about Frame 16-25)/Record start and cause sometimes a buffer overrun or its the change of Intraperiod(change is after Frame 36 but the change may not be realtime - gpu processing delay)

[username7291]

Finally found the Bug. Who has comment such out " //pthread_mutex_lock(&v_mutex);" ??? The callback is called by mmal, while raspimcam.c still runs on its own. Means you have two tasks, both play around with same variables. Of course this fails some times. I confirmed that the callback still change stuff after RaspiMCam set buffering_toggle = 0; For example you prepare a seek position for the stuff in the buffer and after it the buffer is still filled more(this never goes to file), and cause videos get broken and misses Frames. buffering_toggle change while still in the else if(){} condition and cb_wptr change after creating fileSizeCircularBuffer and finally write the buffer to File. If you change v_capturing or buffering_toggle this must protected by a mutex lock, like the whole h264encoder_buffer_callback. Other simplest way is to restore what is commented out, means lock h264encoder_buffer_callback, stop_video, start_video.

[roberttidey]

OK. Well done tracing that down. It certainly makes sense.

I am not familiar with exact details of the why the mutex was commented out but I believe it might have been causing some other issues at the time.

Have you tried uncommenting all the mutex calls?

If that does work which of your other changes remain relevant?

[username7291]

It original locks more stuff than i do right now. The only 3 named Functions and excluding stuff that may take time, like PrintLog/exec_macro. It may removed cause surrounding the whole Functions cause Framedrops. I made a own Mutex Variable cause im not sure if v_mutex is used somewhere else.

-The other prebuffer fix(Bug dump the whole Buffer to file with stuff that is before the fist IFrame) should still required, thats another Bug.

-The testing stuff like removing the intraperiod change should not needed, but for me i leave it in. This seem fixing the very Bad quality Videos sometimes, at about 3000kb/s FHD. I assume the Firmware fails to calculate the requested Bitrate on a Intraperiod change.

-The Buffer size calculation depends on a fixed Bitrate but with Intraperiod change the Bitrate can be lower or much higher. Means sometimes only the halft assumed match into the buffer. Im not sure if overwrite checks work for example if 2 Iframes get overwritten at a Time(different brightness change makes Frames small&big see the issues 621). At all the Buffer system needs changed, its not what the user expect. 1000ms should be 1s and not something else. You get a random amount of frames before a Motion. Doubled buffer and taking a fixed amount of iframes of the Buffer depending on the Intraperiod setting is the simplest for me. For Public this needs to calculated, if user wants 1s use int count = ((long long)cfg_val[c_video_bitrate]/4 * (long long)cfg_val[c_video_buffer]) / 1000; and fps/(Intraperiod*2) is the IFrames/s. Keep in mind FPS can change day/night and too the wrong Intraperiod may not on any Firmware. My is 4.19.88-v7+. You can test it by X-Media-Recode->AVI-Codec Copy. Now load that File into VirtualDub. Go Frame by Frame if you see below [K] its a Keyframe/I-Frame. The Bitrate you can show with MediaInfo.

-The change for the I-Frame request if there is none at 1/4 of the Buffer should be done. The Intraperiod is buggy and gives you all 10 frames a I-Frame if you set it to 5. Means I-Frames get requested unrequired(less Data fits into the Buffer, lower Video quality).

-The Debug 2/3 stuff i have completly removed. Im sure this was for Debugging the Bugs introduced with the removing of the mutex locks. Its not in RaspiVid. Debug 3 is just the output that a overwritten Frame is found, but that too happends on every Capturing Stop, confuses users and say nothing. Debug 2 should check if all I-Frames are Valid, but that function is broken it overflows goes from 0-131071 on invalid Pointers, while only 5 Iframes are in the Buffer. Its not required, if you get invalid I-Frames Firmware is broken or you override it yourself by Bad code. It spams the log/delete it in that way, wears out the SD, may cause Frame Drops.

[roberttidey]

Thanks for the description. Could you possibly zip up your changed source code and upload it. Comparison will make it easier to see the changes and also for the further work.

[username7291]

Seems not that git is compatible with my Browser, i can select a File but it dont appear. Code insert too dont work. I need to upload it extern https://www.filemail.com/d/odvxwtugnhkarme Tryed to make it readable, but i removed stuff for testing if it is a speed issue. This Basicly only takes Videos from the Point of the Motion(no long waiting for the event) with buffer set to 1091(has only a effect on buffersize now). The Intra rate is set to 8 means all 16 Frames a IFrame. Needs to calculated for the Public use. The mutex stuff dont have any bad effect until now, no frame drops seen.

[roberttidey]

Got the file OK. Thanks.

I'll compare and incorporate some changes over the next few days.