Possible causes of missing frames?

[gwappa]

Hi,

I have been trying to write frames to a video file based on callback functions (more or less like https://github.com/TheImagingSource/IC-Imaging-Control-Samples/blob/master/Python/Open%20Camera%2C%20Grab%20Image%20to%20OpenCV/callback.py ) while reading strobe outputs using an external data acquisition device.

The issue I encounter is that "the number of strobe triggers" and "the number of frames in the video file" often differ from each other (the video files have less frames).

In the implementation, the frame index being passed to the callback function is always referred to, so that the frame i be always followed by the frame i+1. Nevertheless, a 1 h video at 160 FPS have ~10 frames less than what it is supposed to have (or 2-3 frames for a 1 h at 30 FPS).

So I would like to hear your opinion on what could be going on in my situation:

• Is it possible that frame drops have nevertheless occurred in the course of grabbing? For example, can one in principle detect frame-drop events by checking the frame indices being passed to the callback functions, or must I not rely on these figures for this purpose?
• Is the callback function possibly dispatched after IC_SuspendLive() or IC_StopLive() are called? If so, is there any way to check how many frames remain to be read from the buffer at the point grabbing stops? (e.g. would it be a sane way to just keep calling IC_GetImagePtr() until a NULL is returned?)

Thanks a lot in advance! Keisuke

[TIS-Stefan]

Hello

The issue I encounter is that "the number of strobe triggers" and "the number of frames in the video file" often differ from each other (the video files have less frames).

Unfortunately this is is an expected behavior of video writing. Hello Keisuke

Is it possible that frame drops have nevertheless occurred in the course of grabbing? For example, can one in principle detect frame-drop events by checking the frame indices being passed to the callback functions, or must I not rely on these figures for this purpose?

Frame drops can happen. The higher the amount of data sent by the camera, the higher is the risc of frame drops. Also writing of frames into a video file is not guaranteed.

Is the callback function possibly dispatched after IC_SuspendLive() or IC_StopLive() are called? If so, is there any way to check how many frames remain to be read from the buffer at the point grabbing stops? (e.g. would it be a sane way to just keep calling IC_GetImagePtr() until a NULL is returned?)

No, Even if the device is closed, the callback function is still assigned to the grabber object, that is referred by the HGrabber pointer. There is also no queue, that contains a list of frames, that are not saved. IC_GetImagePtr() wont be NULL, because there was an image already in this buffer.

I must say, missing 10 frames only at 160 fps in 1 hour video is a very good result.

I am aware, my answer is not really helpful, it acknowledges what you know already. Currently there is no way to check for frame drops in the callbacks, because I simply did not implement the matching functions in the tisgrabber.dll. So I can not be of big help at the moment.

Stefan

[gwappa]

Hi Stefan,

thanks a lot for your detailed answer.

OK, I understand the situation now. I am then going to try on the app side to get rid of any overhead in callbacks, and see if the situation gets better.

Thanks again!

[TIS-Stefan]

Hello

You may remind me on adding these frame counter functions to tisgrabber.dll next year. I started to make a rework of tisgrabber.py anyway, including a getting started tutorial....

Stefan

[gwappa]

Dear Stefan,

actually I have narrowed down what is going wrong with my code in counting frames, and it appears to be because of the way I treat the strobe outputs from the camera.

• When I use the C++ library to check callback stats:
  • FrameQueueSink never reported any frame drops, even though there are differences in the number of frames and the number of recorded strobe outputs.
  • Even when I used NotificationSink and never used the acquired image at all, there were discrepancies between the number of frames and recorded strobe outputs.
• As soon as I call prepareLive(), strobe outputs seem to have been recorded, before I call startLive() (in cases where I use internal triggers to capture frames).
• If I use external triggering, the grabbed videos did not seem to have any discrepancy with the number of recorded strobe outputs (I only tested 30-100 FPS for 3-4 min, though).

Taken together, I presume that the camera runs its capture sequence (and generates strobe outputs) before the library sets up the pipeline to start notifying the user program with any incoming frames. Do you think it is reasonable? If so, is it possible for you guys to update the driver to prevent camera from acquiring images before calling startLive()?

Thanks again, Keisuke

[TIS-Stefan]

Hello Keisuke

I think it is reasonable. prepareLive() starts the camera and sensor, but it pauses the DirectShow filtergraph in the computer. Therefore, you see strobe signals, if the camera is free running.

Stefan