Can't use the usb camera and FP

[chirs123456]

[PINTO0309]

    cam = cv2.VideoCapture(0)
    if cam.isOpened() != True:
        print("USB Camera Open Error!!!")
        sys.exit(0)
    cam.set(cv2.CAP_PROP_FPS, vidfps)
    cam.set(cv2.CAP_PROP_FRAME_WIDTH, camera_width)
    cam.set(cv2.CAP_PROP_FRAME_HEIGHT, camera_height)
    window_name = "USB Camera"
    wait_key_time = 1

    #cam = cv2.VideoCapture("data/input/testvideo4.mp4")
    #camera_width = int(cam.get(cv2.CAP_PROP_FRAME_WIDTH))
    #camera_height = int(cam.get(cv2.CAP_PROP_FRAME_HEIGHT))
    #frame_count = int(cam.get(cv2.CAP_PROP_FRAME_COUNT))
    #window_name = "Movie File"
    #wait_key_time = int(1000 / vidfps)

[chirs123456]

Hi,I have two questions:

1. Can't use the usb camera IR：FP32 + CPU + MP4 is OK But when I use usb camera I have following error: (python3:3641): GStreamer-CRITICAL **: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed I have tried your IR downloaded from google that don't have this problom
2. If I use IR in FP16 and use NCS2, it will have some questions: some boxes's height and width very small, eg. 1e-5

It was my mistake that I just have a issue that only have a title. So what changes should I do? Please help me. Thanking You

[PINTO0309]

I will reply when I have finished my work. Please give me some time.

[PINTO0309]

@chirs123456

But when I use usb camera I have following error: (python3:3641): GStreamer-CRITICAL **: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed

Please tell me the result of the following command. 1.

$ sudo apt-get install v4l-utils

$ ls /dev | grep video*

$ v4l2-ctl -d /dev/video0 --list-formats-ext

or

$ v4l2-ctl -d /dev/video1 --list-formats-ext

or

$ v4l2-ctl -d /dev/videoXXX --list-formats-ext

Example of information on my USB camera.

xxxx@ubuntu:~$ v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2
		Size: Discrete 320x240
			Interval: Discrete 0.005s (187.000 fps)
			Interval: Discrete 0.007s (150.000 fps)
			Interval: Discrete 0.007s (137.000 fps)
			Interval: Discrete 0.008s (125.000 fps)
			Interval: Discrete 0.010s (100.000 fps)
			Interval: Discrete 0.013s (75.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.020s (50.000 fps)
			Interval: Discrete 0.027s (37.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.020s (50.000 fps)
			Interval: Discrete 0.025s (40.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)

If I use IR in FP16 and use NCS2, it will have some questions: some boxes's height and width very small, eg. 1e-5

If you are using tiny-YoloV3, I know that accuracy is very bad. I think that adjustment of anchor at training and revision of model are necessary.

Please refer to the following. https://github.com/mystic123/tensorflow-yolo-v3.git

[chirs123456]

Hi, Thank you for your reply.

1. Yeah, I can use my camera. I set camera_width = 608,camera_height = 608, but the camera support size is I change the camera_width = 640 and camera_height = 480. It's OK
2. When the data_type is FP32, the boxes are OK. But when data_type is FP16 : Sometimes the box's height and width is very small, so in function IntersectionOverUnion(box1,box2), box1.xmin = box1.max, area_of_union=0

for j in range(i + 1, objlen): if (IntersectionOverUnion(objects[i], objects[j]) >= 0.2): objects[j].confidence = 0 above three code will have zero error. This problem is only because the data_type change from FP32 into FP16. The IR is from the weights trained by myself with one category. And I use https://github.com/mystic123/tensorflow-yolo-v3.git to change my weights to frozen_yolov3.pb.

How to address it? Please help me. Thank You

[PINTO0309]

@chirs123456

for j in range(i + 1, objlen): if (IntersectionOverUnion(objects[i], objects[j]) >= 0.2): objects[j].confidence = 0 above three code will have zero error.

Is the code you own as follows? Otherwise the cloned program is outdated. Please try cloning the latest repository.

def IntersectionOverUnion(box_1, box_2):
    width_of_overlap_area = min(box_1.xmax, box_2.xmax) - max(box_1.xmin, box_2.xmin)
    height_of_overlap_area = min(box_1.ymax, box_2.ymax) - max(box_1.ymin, box_2.ymin)
    area_of_overlap = 0.0
    if (width_of_overlap_area < 0.0 or height_of_overlap_area < 0.0):
        area_of_overlap = 0.0
    else:
        area_of_overlap = width_of_overlap_area * height_of_overlap_area
    box_1_area = (box_1.ymax - box_1.ymin)  * (box_1.xmax - box_1.xmin)
    box_2_area = (box_2.ymax - box_2.ymin)  * (box_2.xmax - box_2.xmin)
    area_of_union = box_1_area + box_2_area - area_of_overlap
    retval = 0.0
    if area_of_union <= 0.0:
        retval = 0.0
    else:
        retval = (area_of_overlap / area_of_union)

This problem is only because the data_type change from FP32 into FP16.

Yes. I have recognized problems from long ago. I am also looking for ways to improve it. https://github.com/PINTO0309/OpenVINO-YoloV3/issues/2

btw, In the first place, I know that the accuracy of tiny-YoloV3 is much worse than MobileNet-SSD. https://github.com/PINTO0309/MobileNet-SSD-RealSense.git

mAP is about 20. https://qiita.com/shinmura0/items/967e9eb02724639a7624#_reference-13ddca08cfd77c7ecf3a

[PINTO0309]

I solved the problem of low precision. There was a mistake in the logic of preprocessing and postprocessing.

[chirs123456]

That's good news. Would you please tell me more details? What mistakes and how to address them?

[PINTO0309]

@chirs123456 In the preprocessing and postprocessing, it was a problem that the aspect ratio of the image was not maintained. I reviewed the process to maintain the aspect ratio of the image.

commit d4e7131efab4d24fbd787666337f076aa3b069b9

[chirs123456]

I just try again using the code that you changed. But the result also has same bug. The data_type in FP32 is OK.But data_type in FP16,the box's height and width is very small. eg: print("th={},tw={}".format(output_blob[box_index + 3side_square],output_blob[box_index + 2side_square])) th = -12..4921875 tw=-0.1927490... so height is small and have a series of other problem

[PINTO0309]

@chirs123456 Is it a movement like the figure below?

[chirs123456]

data_type=FP16 the code has problem data_type=FP32 it's a moment like the figure like you

[PINTO0309]

@chirs123456 Sorry. I forgot to fix bugs in that file. I have committed now. However, the accuracy of FP16 is somewhat worse than FP32. commit 3caacfb4e4d396af95f652b28d41b5d82e7d183f

[chirs123456]

I see your changed . But I think that it's useless to force filtering out some boxes data. The true problem is when change the data_type, data is different. Not only weight height, and box's coordinate I plus your changed code it's a problem of model_optimizer? eg:

[PINTO0309]

@chirs123456

it's a problem of model_optimizer?

Yes. That possibility is high. btw, This is the result of FP16.

For the solution, the following will be helpful. https://software.intel.com/en-us/forums/computer-vision/topic/804818 https://software.intel.com/en-us/forums/computer-vision/topic/805370 https://software.intel.com/en-us/forums/computer-vision/topic/805425

[PINTO0309]

Negative slope multiplier was hard coded inside. My next idea was - hmm, maybe some fp16 convertion shenanigans may take place here. What if try to change this constant across all file in LeakyRelu layers. Instead of this number i placed 0.11. And viola! Network started outputting valid results. After some experiments i found out that modification of this slope multiplier in pretty large margins still gives some reasonable results(0.09-0.11), but as long as you have exactly 0.1 (or any number that leads to 0.1 after rounding and convertion) - network outputs some random stuff. And last, but not least: there's actually one critical place in a .xml file where this 0.1 constant value completely breaks network outputs (if you keep 0.1 in other places it still more or less works). That place is right before the convolution layer with 1x1 kernel size, right before YOLO region layer.

<layer id="x" name="LeakyReLU_" precision="FP16" type="ReLU">
            <data negative_slope="0.10000000149011612"/>

↓

<layer id="x" name="LeakyReLU_" precision="FP16" type="ReLU">
            <data negative_slope="0.11"/>