[SUGGESTION] YUV420 to RGB support without cv2

[tvoverbeek]

in #161 cv2 is used to convert YUV420 to RGB. See here (https://github.com/PiSupply/PiJuice/blob/master/MakerKits/Point-shoot-cam/yuv2rgb.c) for a Python module in C which does yuv2rgb conversion. Probably needs some adaptation for Picamera2. This could be useful for e.g. Pi Zero applications which want to use the QPicamera2 widget, but do not want to use cv2.

[davidplowman]

I agree it would be nice to have. Though I struggle a bit to imagine where I might do this without installing OpenCV, do you have any particular use cases in mind?

[tvoverbeek]

The link I posted was for a project to make a small point-and-shoot camera using the V1 or V2 camera and a small 320x240 SPI TFT display with touchscreen as viewfinder. The original project uses the old camera stack and I would like to port it to bullseye using picamera2. Still have to look at how to drive these small TFT displays since I believe the framebuffer code cannot be used any more. Anyway, it is this type of applications camera + small TFT display I have in mind. Should be able to use the original Pi Zero.

[davidplowman]

Sounds like an interesting project. Actually, it might also be an interesting little case study into how you would use Picamera2.

My first reaction for anyone using anything less than a Pi 3, and especially for Pi Zero users, would be that to obtain good (preview) performance you would absolutely have to avoid X-Windows. In fact, on a Pi Zero I think I would go so far as to say: avoid X-WIndows like the plague. That means you'll have to render image usings DRM/KMS - but you'll get drastically better and smoother framerates.

DRM/KMS supports display of YUV420 natively, which certainly removes one reason for converting such images to RGB (this was the principal issue I had in mind already). But might there be others?

I wonder a little bit about thumbnails, though you might expect Python JPEG encoders could create and add them for you. Annoyingly, JPEG really prefers YUV420 but the Python world mostly insists on passing in RGB - I don't know if there are "better" JPEG encoders out there?

There is in fact a YUV to RGB routine here already, and although it's much slower than the OpenCV one, it's probably OK for a small thumbnail when you do a capture. But clearly still lots to explore here!

[popcornmix]

Still have to look at how to drive these small TFT displays since I believe the framebuffer code cannot be used any more.

SPI display can't be directly driven by the HVS, so it's possible on buster you were directing the HVS to an otherwise usused display (e.g. hdmi) then using rpi-fbcp to grab the contents of fake display and copy it to memory buffer SPI is using (effectively mmap of /dev/fb1)

rpi-fbcp won't work with kms. You could use fkms (even on bullseye) but that is deprecated so would only be a temporary solution. But assuing you can convert the camera frame to RGB (in format of SPI framebuffer), then you just need to do the mmap of /dev/fb1 and memcpy the pixels to it. At your resolution that should be straightforward.

[tvoverbeek]

@popcornmix Thanks for confirming what I already suspected. I realized that fbcp will not work any more since it uses mmal functions. Will try out your proposed solution (convert to RGB and memcpy to mmaped /dev/fbx).

[callidus]

I would also find this very helpful. I am building a stop motion setup and would like to capture a request, store the main or raw image to disk and then set the lores image as an overly to ghost the previous frame for my animation. Something along the lines of ...

self.picam2.switch_mode(self.still_config)
request = self.picam2.capture_request()
request.save("main", "test.png")
yuv420 = self.picam2.capture_array("lores")
metadata = request.get_metadata()
request.release()

self.picam2.switch_mode(self.preview_config)
rgb = cv2.cvtColor(yuv420, cv2.COLOR_YUV420p2RGBA)
self.picam2.set_overlay(rgb)

print(metadata)

Unfortunately when I try to import CV2 I get an error due to missing GTK3 deps. I could of course install the missing parts but I would like to avoid deps on GTK (and I guess CV as well) if possible. Having the option to do a conversion (even a slower one) using a built in method would be very useful.

[davidplowman]

In fact there is a YUV420 to RGB conversion routine already, here.

The "problem" with it is that the OpenCV method is much faster, and I don't really know why. All it does is copy the image data into a format in which we can apply a simple numpy matrix multiplication.

So if a few 10s of milliseconds here or there aren't a problem, you could use the method I linked above. Otherwise I'd suggest going with OpenCV. I note there does seem to be gui-less version of OpenCV - pip3 install opencv-python-headless - that might be worth a try.

Alternatively, if anyone knows how to implement a better version, or knows of other Python modules that provide fast conversion methods, I'd be interested to hear about them.

[davidplowman]

So I think I'm going to close this issue, for a few reasons:

1. There is a function already that turns YUV420 into RGB that doesn't rely on OpenCV.
2. The original issue was about a Pi Zero maybe using a QPicamera2 widget. I still can't see any circumstance in which running X-Windows and Qt on a Pi Zero and trying to do a software conversion from YUV to RGB for rendering through the X display stack is going to be anything other than an indescribably terrible experience. So I really don't want to do anything that enables it because I will be flooded with complaints.
3. Avoiding X and using DRM/KMS will be multiple base 10 orders of magnitude better.

If there are any scenarios that I've overlooked please say, otherwise I'll close this in a short while.