Flirpy doesn't correctly detect capture capability (Lepton)

[zoltanszalontay]

Hi, my system used to work fine, but now find_video_device() cannot find the correct camera device.

About the setup:

• Raspberry Pi 4 B+
• Lepton 3.5, PureThermal 2.0
• flirpy updated to 0.1.0

/dev/video* folder looks like this:

crw-rw----+ 1 root video 81, 0 Aug 17 18:45 /dev/video0 crw-rw----+ 1 root video 81, 1 Aug 17 18:45 /dev/video1 crw-rw----+ 1 root video 81, 2 Aug 17 18:16 /dev/video10 crw-rw----+ 1 root video 81, 3 Aug 17 18:16 /dev/video11 crw-rw----+ 1 root video 81, 5 Aug 17 18:16 /dev/video12 crw-rw----+ 1 root video 81, 4 Aug 17 18:16 /dev/video2

I have a pi_camera and a PureThermal 2.0 attached.

The error message I get. Pls note that find:video_device() finds device #1. VIDEOIO ERROR: V4L2: Could not obtain specifics of capture window. VIDEOIO ERROR: V4L: can't open camera by index 1 /dev/video1 does not support memory mapping

Manually setting the device ID to 0 I still get the camera works fine.

Any idea is appreciated, Zoltán

[zoltanszalontay]

Attaching the output of for d in /dev/video* ; do echo $d ; v4l2-ctl --device=$d -D --list-formats ; echo '===============' ; done video.txt

It is clearly seen that device #0 has video capability 0x200, but for some reason device #1 is selected, which then fails to support capture.

[jveitchmichaelis]

@zoltanszalontay I've made a quick fix here. The issue is that we check whether the result from grab is None, but in fact grab returns a tuple with a success status and the image, e.g. (bool, np.array). So checking the result always returned true. This was already corrected in the Boson class.

https://github.com/LJMUAstroecology/flirpy/commit/8da2a433301b357d53fcbe8f0ce65562c2cec987

We can probably filter on ID_V4L_CAPABILITIES == ":capture:"?

Can you try the following:

            for device in video_devices:
                udev = pyudev.Devices.from_path(context, device)

                try:
                    vid= udev.properties['ID_VENDOR_ID']
                    pid = udev.properties['ID_MODEL_ID']
                    caps = udev.properties['ID_V4L_CAPABILITIES']

                    if vid.lower() == "1e4e" and pid.lower() == "0100" and ('capture' in caps):
                        dev.append(int(device.split('video')[-1]))
                except KeyError:
                    pass

That's a cleaner way of doing this if it works, as we can check the capability directly. I have a PT2 somewhere around here, but I can't find it right now :)

[zoltanszalontay]

Hi, thanks for the suggestion. Unfortunately, it still doesn't work for me.

The dev array is [1, 0] so it finds both Lepton device drivers. Pls note that /dev/video0 and /dev/video1 both have the capability ':capture:'. I printed them out, see below -->. The device order is 1 and 0. After that it tries to capture from device1 and cam = cv2.VideoCapture(d + cv2.CAP_V4L2) fails.

This is how /dev/video1 and /dev/video0 are collected:

vid: 1e4e pid: 0100 caps: ':capture:' dev: [1] vid: 1e4e pid: 0100 caps: ':capture:' dev: [1, 0]

Obviously, it would work if you checked the devices in reverse order so video0 would become the first, but I do not know why the device driver for video1 cannot capture if its capability says so.

[zoltanszalontay]

Addendum:

As I said, cam = cv2.VideoCapture(d + cv2.CAP_V4L2) fails and generates error messages on screen, but returns

data: (False, None)

so IMHO apart from None, those return values should be ignored too.

[jveitchmichaelis]

OK, then it should be fixed here:

https://github.com/LJMUAstroecology/flirpy/commit/8da2a433301b357d53fcbe8f0ce65562c2cec987

Can you try that before I push it to pypi? That fix checks both return values from grab.

Obviously, it would work if you checked the devices in reverse order so video0 would become the first, but I do not know why the device driver for video1 cannot capture if its capability says so.

Both cameras are checked, but there was a bug in the condition as you note in your addendum. Maybe a bug in PT2 though? My webcam displays two devices, but only one reports it can capture. It's a big ugly, but actually taking an image is probably the most robust way of doing this. I couldn't see anything else obvious in udev that could be used to differentiate. We could resort to parsing v4l2-capture, but I don't think that's any better than just trying to take an image and checking to see if it fails.

[zoltanszalontay]

It's weird.

Both video1 and video return data: (False, None) so the fix above does not work in my case. It seems my camera returns bad meta info, but video0 can execute the capture at cca. 8.7fps.

I do not know if it is important in this case, but I'm using a Raspberry Pi 4B+ and the camera is plugged into a USB2 slot, because USB3 is occupied by a Google Coral Edge TPU and if I mix the two kinds of USB devices, the slower will make the USB3 hub slower.

In theory, the PureThermal requests 12mbps and should get it, but I do not know if the Pi camera, which is connected to the same hub AFAIK, makes any trouble.

All in all, even if my PureThermal is working fine if I select /dev/video0 manually, I always get a "VIDIOC_STREAMON: No space left on device" error message, which is just a bad way of saying no USB bandwith could be allocated.

[jveitchmichaelis]

Both video1 and video return data: (False, None) so the fix above does not work in my case. It seems my camera returns bad meta info, but video0 can execute the capture at cca. 8.7fps.

Are you sure flirpy is actually checking the right video devices?

Otherwise this is weird as you say, because why would video0 fail during enumeration, but work if you pass 0 manually? There's no real difference in how the camera is opened.

[zoltanszalontay]

I dont't get it either. I can check it tomorrow morning the soonest. I'll report back.

Thanks a lot, Z

[jveitchmichaelis]

One option (which doesn't work in Flirpy yet!) would be to use the SPI breakout board with the Pi? That would avoid your bandwidth issue perhaps.

You can use usbtop to sniff what's going on by the way, I've found it very useful in the past.

Or if you're using the Coral for the Lepton imagery, you could just run it over USB2? If your max fps is only 9Hz then you're not going to bottleneck the accelerator which is receiving images at like 300x300 max (and a quarter of that if you're using a Lepton).

[zoltanszalontay]

I have a couple of SPI BB and I don't like them. SPI BreakOut boards use a very high synchronous clock that a Raspberry simply cannot provide due to its lack of RT clock. Many people, as me, receive broken frames every 3-5 seconds. Not a reliable product. That's why I switched to PureThermal USB.

You're right about running Coral on USB2. iIn a USB3 slot it can reach 30fps with ssd_mobilenet_v2_face_quant_postprocess_edgetpu.tflite for face detection.

My plan was to run face detection on Coral, face tracking on CPU and run PureThermal USB processing in a separate thread @ 9fps.