Error when calling gocv.IMEncode inside a loop

[magallardo]

Description

Hello, I am trying to get the bytes from an image coming from an rtsp stream and I used the IMEncode function to do that. Everything works fine for a few times, but it throws the error below after a few iterations. Note that the sample works fine without the call to IMEncode.

outbytes, _ := gocv.IMEncode(gocv.JPEGFileExt, img)

Please advise if there is a better way to get the bytes for the image.

libc++abi.dylib: terminating with uncaught exception of type cv::Exception: OpenCV(4.5.0) /tmp/opencv-20201123-26930-m95s1u/opencv-4.5.0/modules/imgcodecs/src/loadsave.cpp:934: error: (-215:Assertion failed) !image.empty() in function 'imencode'

SIGABRT: abort
PC=0x7fff6b89833a m=0 sigcode=0
signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x40ab650, 0xc00005de60, 0x400f5b8)
        /usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc00005de30 sp=0xc00005ddf8 pc=0x400623b
gocv.io/x/gocv._Cfunc_Image_IMEncode(0x5be36530, 0x14689390, 0x0, 0x0)
        _cgo_gotypes.go:2328 +0x4d fp=0xc00005de60 sp=0xc00005de30 pc=0x40a665d
gocv.io/x/gocv.IMEncode.func2(0x5be36530, 0xc00005df20, 0x0, 0x0)
        /Users/mag/workspace/go/pkg/mod/gocv.io/x/[email protected]/imgcodecs.go:204 +0x64 fp=0xc00005de98 sp=0xc00005de60 pc=0x40a78e4
gocv.io/x/gocv.IMEncode(0x40f675c, 0x4, 0x14689390, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/mag/workspace/go/pkg/mod/gocv.io/x/[email protected]/imgcodecs.go:204 +0xaa fp=0xc00005df10 sp=0xc00005de98 pc=0x40a6efa
main.main()
        /Users/mag/workspace/src/deviceservicertsp_gocv/cmd/main.go:16 +0xda fp=0xc00005df88 sp=0xc00005df10 pc=0x40a7dba
runtime.main()
        /usr/local/go/src/runtime/proc.go:203 +0x212 fp=0xc00005dfe0 sp=0xc00005df88 pc=0x4034a72
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x405f1b1

rax    0x0
rbx    0xe077dc0
rcx    0x7ffeefbfd0f8
rdx    0x0
rdi    0x307
rsi    0x6
rbp    0x7ffeefbfd120
rsp    0x7ffeefbfd0f8
r8     0x7ffeefbfcfc0
r9     0x7d
r10    0xe077dc0
r11    0x246
r12    0x307
r13    0x3000000008
r14    0x6
r15    0x16
rip    0x7fff6b89833a
rflags 0x246
cs     0x7
fs     0x0
gs     0x0

Steps to Reproduce

1. I just added a couple of lines of code to the gocv sample.

func main() {
	webcam, _ := gocv.OpenVideoCapture("rtsp://xxxxxxxxxxxxxxxxxxxxxxxxxxx")
	window := gocv.NewWindow("Hello")
	img := gocv.NewMat()

	for {
		webcam.Read(&img)
		outbytes, _ := gocv.IMEncode(gocv.JPEGFileExt, img)
		if outbytes == nil {
			fmt.Printf("outbytes is nil")
		}

		window.IMShow(img)
		window.WaitKey(1)
	}
}

Your Environment

• Operating System and version: MacOS Catalina 10.15.6
• OpenCV version used: 4.5.0
• How did you install OpenCV? homebrew
• GoCV version used: v0.25.0
• Go version: go1.14.3 darwin/amd64
• Did you run the env.sh or env.cmd script before trying to go run or go build? No

[deadprogram]

You should probably check if img.Empty() before calling gocv.IMEncode() would be my suggestion.

Hope that helps!

[magallardo]

@deadprogram That is not the issue as the ima is shown in the window fine if I remove the ImEncode call. I also tried putting an if condition to call ImEncode only in !img.Empty() but still get the same error. It works for about 50 iterations and then throws the error. I think there might be some memory leaks.

Thanks

[deadprogram]

What are you actually trying to do here? The code does not make much sense to me as written?

[magallardo]

@deadprogram The sample code is just to demonstrate the error. I am trying to get the byte data from each image frame in the video stream to send to another applications.

Thanks.

[deadprogram]

Check out https://github.com/hybridgroup/gocv/blob/release/cmd/mjpeg-streamer/main.go which sounds like it does something like what you are looking for.

[magallardo]

@deadprogram I have looked at many samples and they are following the same pattern. This is working fine if I just do for a few frames. However, the issue is when I execute the loop for more than ~50 times (the code failure is not always at the same number of iterations). If I run for a few iterations, there is not issue. That is why I think there is some issue with freeing/allocating memory.

Thanks

[deadprogram]

Well, I would investigate the streaming source of video data. The code here https://github.com/opencv/opencv/blob/master/modules/imgcodecs/src/loadsave.cpp#L934 is pretty clear that it does not think that there has been any image passed into the OpenCV function.

[magallardo]

@deadprogram Thanks. I will make sure I put all the checks in the code to avoid calling the method, when image is empty. I tried that already but will add some more checks.

Thanks

[magallardo]

@deadprogram I put some extra checks and it seems like the source of the problem is the webcam.Read(&img) statement. It seems like this is returning an error.

I added the following code and now I am getting the following:

for {
		fmt.Printf("Processing image: %d\n", i)
		if ok := webcam.Read(&img); !ok {
			fmt.Printf("RTSP Device closed \n")
			return
		}
...
...

[h264 @ 0x35843200] error while decoding MB 4 61, bytestream -11 RTSP Device closed

Thanks for your help. Now I need to figure it out why is this happening when I call the gocv.IMEncode function. If I comment that call, then the loop executes without any issue.

Thanks

[magallardo]

@deadprogram I put some extra debugging statements, and it seems like the issue is actually happening in the IMEncode function. I have been seeing some logs in the console that come from the IMEncode function and every time after I see this messages, the call to webcam.Read fails.

Following is the output after I put some log statements before and after the IMEncode call.

Processing image: 39 Calling IMEncode [h264 @ 0x14843c00] left block unavailable for requested intra4x4 mode -1 [h264 @ 0x14843c00] error while decoding MB 0 34, bytestream 889 Completed IMEncode Processing image: 114 RTSP Device closed

[deadprogram]

You could try having 2 Mat, one that you use to capture into, and then another that you use to encode from. Untested examples based on your example would be something like this:

func main() {
	webcam, _ := gocv.OpenVideoCapture("rtsp://xxxxxxxxxxxxxxxxxxxxxxxxxxx")
	window := gocv.NewWindow("Hello")
	img := gocv.NewMat()
        cpy := gocv.NewMat()

	for {
		webcam.Read(&img)
                img.CopyTo(&cpy)
		outbytes, _ := gocv.IMEncode(gocv.JPEGFileExt, cpy)
		if outbytes == nil {
			fmt.Printf("outbytes is nil")
		}

		window.IMShow(img)
		window.WaitKey(1)
	}
}

[magallardo]

@deadprogram Even after I do the copy, I am still getting the error when the IMEncode function is called. If I just do the copy and comment the ImEncode, everything is still working fine. Definitely there is something in the IMEncode function that is corrupting the memory that after a few executions, the webcam.Read function fails. Following is the log in the console when I run with the copy.

Calling IMEncode [h264 @ 0x14870e00] left block unavailable for requested intra mode [h264 @ 0x14870e00] error while decoding MB 0 43, bytestream 1323 Completed IMEncode

Thanks for your time and help.

[golubaca]

I am just curious why nobody checks for an error returned by IMEncode. Always check for errors,never ignore. And conversion is done by the ffmpeg, thats why h264 errors.

[magallardo]

@golubaca Unfortunately the IMEncode function is not returning any error. There is something being corrupted and it manifests later when the webcam.Read operation is called and fails. Following is the final sample code to reproduce the issue.

func main() {
	webcam, err := gocv.OpenVideoCapture("rtsp://xxxxxxxxxxx")
	if err != nil {
		fmt.Printf("Error opening RTSP device\n")
		return
	}
	defer webcam.Close()

	window := gocv.NewWindow("Hello")

	img := gocv.NewMat()
	dcimg := gocv.NewMat()
	defer img.Close()
	defer dcimg.Close()
	i := 0

	for {
		fmt.Printf("Processing image: %d\n", i)
		if ok := webcam.Read(&img); !ok {
			fmt.Printf("RTSP Device closed \n")
			return
		}

		if img.Empty() {
			continue
		}

		img.CopyTo(&dcimg)
		fmt.Printf("Calling IMEncode\n")
		outbytes, err := gocv.IMEncode(".jpg", dcimg)
		fmt.Printf("Completed IMEncode\n")
		processImage(outbytes)

		if err != nil {
			fmt.Printf("Error in IMEncode: %v\n", err)
		}
		i++

		// window.IMShow(img)
		window.WaitKey(1)
	}
}

As indicated in the small capture of the logs, the error is not returned on the function call. The log is just logged internally in the IMEncode call

Processing image: 38 Calling IMEncode [h264 @ 0x1484c200] error while decoding MB 85 48, bytestream -27 Completed IMEncode Processing image: 39

Thanks

[golubaca]

Ok, first of all, there is no need to use two Mats because when passed to IMEncode, new Mat is created. We are not passing pointer to this Mat, and execution is stopped until this function finishes. The real problem here is not IMEncode itself, but the stream. As you can see, in the imgcodecs.go we are using passed image, and passing Mat's underlying object pointer to the opencv imencode. As you can see, in loadsave.cpp file in opencv repo, line 934, which you already get from error in first comment on first line, says CV_Assert(!image.empty()). This check if Mat is empty. Whats interesting here is the logs, [h264 @ 0x1484c200] error while decoding MB 85 48, bytestream -27 if from ffmpeg, not opencv. It happens when frame cant be decoded or have another issues. After processing frame in Read function, you have Mat object, so there is no need for h264 after that. Problem is, like in first comment, you get a panic because of assertion on opencv part, but now you just get an errors from ffmpeg. I often get output like this, but no issues at all. Can you load different stream and check that this behaviour continues?

[urjitbhatia]

@magallardo This looks like your rtsp endpoint isn't streaming fast enough or returning some corrupted frames. You might have to tweak the rtsp server settings like buffer size or see some other hacks from the related links like fiddling with nice values. It looks like your waitKey(1) is just too quick and your stream isn't pushing data that fast. You're trying to capture packets at 1 per ms. Try 30 ish to get a 30 fps frame rate?

possibly related:

• https://superuser.com/a/1525996/758250
• https://stackoverflow.com/questions/50063707/ffmpeg-rtsp-error-while-decoding-mb

[golubaca]

@urjitbhatia using WaitKey has no impact at framerate if it is lover that waitkey. It has when waitkey is higher than actual framerate. But I agree with corrupted stream. Also, network can be an issue sometimes. If stream is sent over UDP it cant guarantee data delivery, and frames get corrupted. You can try with local files, if this issue is persistent with multiple local files, than we have a problem. I tried with multiple sources and never encounter issue like this.

[magallardo]

@golubaca I really think there is an issue with the encoding. As I mentioned in my original post, if I remove the Encoding line, then there is not issue at all. The loop executes 1000s of times. As soon as I add the Encoding, the error is thrown after about 50 iterations. I also removed displaying to a window to minimize to error points.

func main() {
	webcam, _ := gocv.OpenVideoCapture("rtsp://xxxxxxxxxxxxxxxxxxxxxxxxxxx")
	img := gocv.NewMat()

	defer img.Close()
	defer webcam.Close()

	for {
		webcam.Read(&img)
		outbytes, _ := gocv.IMEncode(gocv.JPEGFileExt, img)
		if outbytes == nil {
			fmt.Printf("outbytes is nil")
		}

		time.Sleep(5 * time.Second)
	}
}

Thanks

[golubaca]

@magallardo did you try with local file as I mention? I tried several IP cameras,multiple local fils, at multiple dfferent OS-es and didnt get a single error you're mentioned.

[magallardo]

@golubaca I will try with local files. But my question is, why the stream works fine and displays on the window fine if I don't call the Encoding operation? If there are issues with the stream, displaying on the window will also have issues.

Thanks

[urjitbhatia]

@golubaca ah interesting, I thought that calling read in a loop too fast was throwing those errors from ffmpeg saying it doesn't have anything in the buffers yet. Good to know that it's an idempotent call if the underlying buffer isn't ready yet.

@magallardo I saw this line in your comments left block unavailable for requested intra4x4 mode -1 and I wonder if that could be packet loss. Try consuming from your stream with just ffmpeg for a while and see if it errors? You can write the stream to a file like ffmpeg -i rtsp://@<yourstream> -acodec copy -vcodec copy <outputlocation>.mp4 or even play the stream directly with ffplay. That might give you clues about the underlying issues with the stream and isolate the problem wrt gocv.

[topeolufe]

I'm also getting a similar issue. @magallardo were you able to resolve this issue?