aravissrc reuses for capture buffer sent forward in GStreamer pipeline

[kohtala]

trafficstars

Describe the bug Capture from camera may overwrite buffer that is still being used in the GStreamer pipeline.

To Reproduce Reduce the aravissrc num-buffers and set hight fps. Add queues early in pipeline to avoid copy and add delay in the pipeline after queue.

End of pipeline will receive corrupt or newer frames than expected.

I've not confirmed this by testing, I just read the source. There is a chance something prevents this problem, but I did not see it.

Expected behavior Frames reach end of pipeline intact.

Camera description:

• Any manufacturer
• Any model
• Any interface

Platform description:

• Aravis version ARAVIS_0_7_4-33-g7337399
• OS: Any
• Hardware any

Additional context The gst_aravis_create calls gst_buffer_new_wrapped_full to create a GstBuffer referencing same memory as used by the ArvBuffer. It then calls arv_stream_push_buffer to return the buffer to ArvStream and returns the GstBuffer to pipeline thus giving the same buffer memory to both.

I'm fairly inexperienced with Aravis and GStreamer, but to me it seems a fix to this issue could also address other related issues:

• gst_aravis_create keeps the GstAravis locked for the duration of arv_stream_timeout_pop_buffer call. Would need some other way to allow releasing the lock for other processing during this time to help low fps cases. This could be a simple increased ref on ArvStream during this call and checking we still have same stream on the GstAravis when again locked. I just did the easy and safe thing when I added the locking, not the best.
• GStreamer can negotiate buffer allocation with the sink element that will use the buffer. If there is for example GPU processing, it could map the buffer from GPU memory to avoid copying it later. I've yet to look how the GstBufferPool could be integrated to the ArvStream.
• I suspect the copy done with comment "Gstreamer requires row stride to be a multiple of 4" would not be necessary, if the buffer had GstVideoMeta telling the stride. It only defaults to 4 on video/x-raw if not told otherwise. This meta would be assigned to the GstBufferPool.
• The use of num-buffers property differs from the one in the base class. I'd rather see the expected use of num-buffers kept and if this is still needed after buffer rewrite use some other propery.

I do not see these as critical and may not have time to work on them.

[EmmanuelP]

Yes, that is something that is taken care of in arv-viewer. When the aravis buffer is wrapped in a gstreamer buffer, we register a callback who has the responsibility of putting back the aravis buffer in the stream receiving thread buffer pool.

https://github.com/AravisProject/aravis/blob/bc9b17b80aa772c60d92d7cfa0e0cb41b0cd4b9b/viewer/arvviewer.c#L330-L332

https://github.com/AravisProject/aravis/blob/bc9b17b80aa772c60d92d7cfa0e0cb41b0cd4b9b/viewer/arvviewer.c#L264-L288

[kohtala]

Thanks. I did not notice that. I thought about that kind of solution for GstAravis though, but then looked at GStreamer memory management and found the GstBufferPool. Search in GStreamer sources tells there are a number of GstBufferPool classes for different video surfaces and codecs (GL, XVimage, XImage, KMS, Framebuffer, DirectFB, V4L2 buffer and codec) where using their pool could avoid a copy. A solution towards using them would mean cheaper HW could do more. Less heat?

But there are new things that need studying when I don't know about them. It could mean for example that the ArvStream would need to make a call on the buffer to map the buffer into process address space before access.

[EmmanuelP]

At this point, you know more about gstreamer than me. I learned just the basics in order to get the plugin output a video.

Your interest in the gstreamer plugin is very appreciated, and any improvements in performance or stability is welcome.

[kohtala]

Thank you. Appreciation goes both ways. Your project and the contributions of people have helped me and I'm glad to be able to return the favor.

This is interesting. As I have clients and other duties also I can not help as much as I'd like. I hope I'll be able to provide a fix to this. Currently I need only low framerate and do not have a real problem with this, but if large resolution cameras at high framerate will be needed I certainly will need to return to this to see if it helps avoid allocating frames needlessly and for reliability. There will be some processing in the pipeline, but I am not sure if it'll be able to use GPU.

[alicrop]

Currently suffering from this issue while using gstreamer. Would increasing the buffer numbers be the easiest mitigation?