non-standard Grab() function - how do I flush the buffers?

[marktheunissen]

Hi all, is there a reason why the implementation of VideoCapture.Grab() differs so much from opencv core?

void VideoCapture_Grab(VideoCapture v, int skip) {
    for (int i = 0; i < skip; i++) {
        v->grab();
    }
}

There's no mention of a "skip" parameter.

I'm looking for a way to flush out the pending buffers and stumbled across this function call which appears to be what I need but doesn't seem to work as expected. Basically, I need to always get the most recent video frame whenever I call .Read(), and don't care if any are dropped. So I thought I could call .Grab(50000) and it would just flush out all pending / queued frames and then I could call .Read() after that to then get the most recent one.

However it looks like Grab(50000) will actually wait for 50000 frames to come in... any idea if this is because of the non-standard way it's been implemented in GoCV or if OpenCV actually cannot do this?

Thanks for the all the work you are all doing on this project.

[deadprogram]

Hi @marktheunissen looking at this, I cannot remember why it I implemented in this way. Seems like an error on my part, probably should change to match the "real" api. I'd be happy to receive an PR with this change.

However, according to the OpenCV docs, grab() should return 0 if no frame is grabbed, and 1 if a frame is grabbed, so it should not block.

https://docs.opencv.org/3.4.2/d8/dfe/classcv_1_1VideoCapture.html#ae38c2a053d39d6b20c9c649e08ff0146

Have you tested this with your hardware setup to see what it does?

[marktheunissen]

Interesting, thanks for getting back - I did some digging and it looks like the Gstreamer implementation of the VideoCapture->Grab() method will call gst_app_sink_pull_sample which does block, and breaks the general advertised API contract of the VideoCapture interface.

There's another function called gst_app_sink_try_pull_sample which takes a timeout and returns if it doesn't get a sample in time.

I'll have to keep poking at things. I essentially need to try figure out how to get my pipeline to be as close to realtime as possible and stop buffering data (eventually it crashes with OOM). appsink indicates that it can drop buffers, and I've reviewed that code and it all seems correct, but gstreamer is such a rabbit hole...

https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-appsink.html#gst-app-sink-try-pull-sample

[deadprogram]

I have been wanting to do some more advanced things with gstreamer myself, specifically for decoding h264 video. So I find your experiments especially interesting.

[marktheunissen]

This pipeline works well for me (opening it through OpenCV):

rtspsrc location=rtsp://192.168.1.1/path/to/stream drop-on-latency=true do-retransmission=false latency=0 ! decodebin ! videoconvert ! appsink max-buffers=1 drop=true

That's a security camera pushing out h264 video. It all works fine until a few seconds in when it starts drifting, somewhere there is a buffer filling up. When I get the next frame and compare the frame's timestamp (the time is overlayed on the image itself), it can grow to be 2-3 minutes behind.

I don't think at this point that the issue is OpenCV or the appsink element, working my way backwards through the pipe's code to figure out where it's buffering. I suspect rtspsrc will be the culprit. Here's someone reporting similar issues: http://gstreamer-devel.966125.n4.nabble.com/Buffer-PTS-drifts-into-the-future-td4680148.html

[marktheunissen]

Ok I basically solved all my issues by reducing any one of the source video's frame rate, bitrate or resolution.

720p works great, no delay, stable and smooth. I think the issue is that while the RPi can technically decode h264 at 1080p, the minute you start shifting that data around on the shared bus (USB/eth0), it can have unpredictable results when the data is large enough. I'm pulling a bit stream and then moving frames over USB to the NCS. This is just a hunch - I no longer suspect any particular software of being the issue since it works flawlessly at lower bandwidths.

The RPi decodes the 264 on the OMX GPU, since the decodebin will select that plugin as the only viable 264 decoder.

https://raspberrypi.stackexchange.com/questions/44433/does-wifi-on-the-pi-3-reduce-the-ethernet-bottleneck-when-using-usb-device

Gstreamer is a weird beast, it's amazing when it works, but it can be quite frustrating figuring out where the pipeline has gone wrong. I found turning on debug logs, reading the source of the plugins and really spending the time to understand all the various caps like pixelformat, etc, greatly helped. Once it's running though, it's great.

[feipengheart]

Hello, I have the same problem: How can Grab jump to the latest frame without blocking