TheImagingSource
Random Hang on GStreamer Pipeline When Repeatedly Opening and Closing Camera in tiscamera
I am encountering some peculiar technical issues while working with tiscamera in conjunction with gstreamer. I am seeking advice on potential solutions or methods for further debugging.
When using the tiscamera driver, if I repeatedly open and close the camera in the application, there is a random chance that the gstreamer pipeline would hang when it's being closed (when the state is set to null).
Initially, I thought it was an issue with my software, but I eventually traced it back to tiscamera. I've forked the repository and added some debug messages, and here are the results:
Here's a code snippet from the version of tiscamera that I forked today: link
For a normally functioning camera, after being "done", it will proceed with actions like freeing memory for components. But when I switch the camera on and off multiple times (sometimes after 5-6 times, but often after 100 or even hundreds of times), this occurs:
After "done", it just gets stuck, and I can't proceed with the next step. (I want to emphasize here that I speculate after stop(), it should immediately go to dispose(). But I don't know why after stop() is fully executed, it can't go to dispose().)
I initially thought this problem was caused by operating multiple cameras simultaneously (originally 9), but later when I used only 1 camera, the same issue occurred.
I'm not sure if this is a problem with the tcam plugin or with gstreamer. However, in my entire gstreamer pipeline, all components can be properly closed except for the tcam component. When trying to close the tcam component, it hangs as described above.
I've looked through the project's related issues, but I haven't found any similar content (the issues raised by others on setState(NULL) getting stuck seem different), so I'm raising this question here.
Computer used OS: Ubuntu 22.04.2 Architecture/Platform: x86/64 Camera model: 33UX287 or 33GX287 or 33UX273, all happened tiscamera version: Latest build from source (1.1.0)
Driver code snippet
class TisPipelineGStreamer(private val settings: GstTisSettings) : TisPipeline {
private var pipeline: Pipeline? = null
private var appSink: AppSink? = null
override fun start() {
if (!Gst.isInitialized())
Gst.init()
pipeline = Pipeline("pipeline")
// document: https://www.theimagingsource.com/en-us/documentation/tiscamera/tcam-gstreamer.html#tcambin
val tcambin = ElementFactory.make("tcambin", "source").apply {
set("type", settings.type)
}
val capsfilter = ElementFactory.make("capsfilter", "filter").apply {
set(
"caps", Caps.fromString(
"video/x-raw, " +
"format=BGRx, " +
"width=${settings.width}, " +
"height=${settings.height}, " +
"framerate=${settings.framerate}/1, " +
"binning=1x1, " +
"skipping=${settings.skipping}"
)
)
}
val queue = ElementFactory.make("queue", "queue").apply {
set("leaky", 1)
set("max-size-time", 0)
set("max-size-bytes", 0)
set("max-size-buffers", 0)
}
appSink = AppSink("appsink").apply {
set("drop", true)
set("emit-signals", true)
set("enable-last-sample", true)
set("max-buffers", 5)
set("sync", false)
}
arrayOf(tcambin, capsfilter, queue, appSink).also {
pipeline!!.addMany(*it)
Pipeline.linkMany(*it)
}
tcambin.set("serial", settings.serial)
startPipeline(pipeline!!)
}
override fun pullSample(): Frame {
val bytes = copyBytesFromSample(appSink!!)
return FrameBasic(settings.width.toDouble(), settings.height.toDouble(), bytes)
}
override fun stop() {
Gst.getExecutor().submit {
println("Stopping ${settings.serial}")
pipeline?.stop()
}.get()
}
private fun startPipeline(pipeline: Pipeline) {
pipeline.ready()
pipeline.play()
val bus = pipeline.bus
bus.connect(Bus.EOS { source: GstObject? ->
println("EOS")
Gst.quit()
})
bus.connect(Bus.ERROR { source: GstObject?, code: Int, message: String ->
println("ERROR $message")
Gst.quit()
})
}
private fun copyBytesFromSample(elem: AppSink): ByteArray {
val sample = elem.pullSample()
val buffer = sample.buffer
val byteBuffer = buffer.map(false)
val bytes = ByteArray(byteBuffer.remaining())
byteBuffer[bytes, 0, bytes.size]
releaseMemory(buffer, sample)
return bytes
}
private fun releaseMemory(buffer: Buffer, sample: Sample) {
buffer.unmap()
sample.dispose()
}
Hello We suspect, there is a handle / memory leak in Gstreamer which creates an issue after 471 cycles. This happens without any software from us too. Also a customer found that GStreamer floods the bus with many bus message, thus the bus should be flushed C++: gst_bus_set_flushing(bus, TRUE);. That should be called each time the pipeline is set to STATE_NULL. I think, there is something similar in Python. You may try this.
Stefan
Hello, I've tried adding auto flush bus, but it still seems to have the problem. It still hangs at the original stop point.
Here is my updated code:
override fun stop() {
Gst.getExecutor().submit {
println("Stopping ${settings.serial}")
pipeline?.bus?.setFlushing(true)
pipeline?.stop()
}.get()
}
private fun startPipeline(pipeline: Pipeline) {
pipeline.ready()
pipeline.play()
val bus = pipeline.bus
println(pipeline.autoFlushBus)
pipeline.autoFlushBus = false
bus.connect(Bus.EOS { source: GstObject? ->
println("EOS")
Gst.quit()
})
bus.connect(Bus.ERROR { source: GstObject?, code: Int, message: String ->
println("ERROR $message")
Gst.quit()
})
}
I've also tried executing flush() after stop(), but the issue still persists.
Additionally, I tried replacing src with videotestsrc to see if the issue is related to gstreamer. However, it seems to be able to run smoothly over 900 times (until I manual stop the process) (if tcambin is put in, it can only run up to 400 times at most before an issue occurs).
override fun start() {
if (!Gst.isInitialized())
Gst.init()
pipeline = Pipeline("pipeline")
val videoTestSrc = ElementFactory.make("videotestsrc", "videotestsrc")
... The rest of the code is as provided above
This issue feels quite tricky. I wonder if you could give any insight into what operations gstreamer would perform between gst_tcam_buffer_pool_stop and gst_tcam_buffer_pool_dispose as mentioned above? That might help me to dig deeper and print more debug messages.
(https://github.com/lulu2002/tiscamera/blob/267350ca6507742bc7893432468ced2e64dc32b4/src/gstreamer-1.0/tcamsrc/gsttcambufferpool.cpp#L420-L433)
Hello
I am not the tiscamera programmer, thus I can not give any insights. But the source code of tiscamera is available at https://github.com/TheImagingSource/tiscamera
I am not sure about what causes this error situation. Before writing wrong stuff, I will wait for my colleague who has more experiences with GStreamer.
I wonder, whether you get the same situation, if you use v4l2src instead of tcambin and the USB camera. If the situation shows up there too, then it is a GStreamer error. Could you please try this and let me know the result?
Stefan
