YCbCr (YUYV)

[raymanfx]

I am currently writing a cross-platform image streaming I/O library mainly focused on webcam capture. Most webcams do not provide RGB buffers, but almost all of the cheap ones usually provide YUYV. While I could write my own YUYV -> RGB routines and handle conversion in the library, I wanted to ask how you feel about supporting YUV family formats in this crate.

Draft

8 and 16 bit YUV (single-planar aka interleaved) support would look just like the existing Rgb struct: pub struct Yuy2<T: Primitive>(pub [T; 4]);

Notes:

• YUYV == YUY2
• 4 components describe a macropixel, which is two image pixels

I think it would make sense to focus on interleaved formats in the first step and add planar formats later on.

[tylerhawkes]

I would love to have YUV support. I currently get frame buffers from a camera in the yuv_nv12 format and having support for them here would allow me to stop depending on opencv all together.

[raymanfx]

I evaluated adding YUV support to this crate, but found that its internal architecture is too inflexible. Adding a new format requires touching many internal structures (cannot be done in an external crate) and there's no multi-planar API at all afaik.

I started developing a new crate for foreign function image conversion: https://github.com/raymanfx/ffimage. The design is flexible and I have a second subcrate to handle YUV formats explicitly (not on GitHub right now, but I can push it). Conversion is parallelized by Rayon, so it's quite efficient.

I was focussing on the packed API for now, but if you'd like to help, we can flesh out the multi-planar API and add support for your usecase.

[197g]

Indeed, the design of ImageBuffer is quite dated and large portions of it haven't been changed motivated by compatibility and effort. The most glaring issues are that it hadn't been able to consider safe byte casting when it was first implemented and retro-fitting is (as you've observed) almost impossible.

For what it's worth, our own experiments with a new buffer live in image-canvas. I'm cooking up one large change to the internals that make layouts a lot more flexible. You should soon be able to in-place reinterpret between a macro pixel layout and the channel view of it. Also between the internal bytes and your own Rgb type which will solve the necessity to agree on a common struct by avoid it :) The same buffer could also fit Yuv420 and Linux's drm FourCC descriptors without reallocation. Fast conversion would then be a small step away (or live in user crates). Somehow magically it is almost entirely free of unsafe, safe for two DST casts that are not expressible in Rust right now.

[fintelia]

One question I have is what you actually want to be able to do with YUV data? Are you hoping to do image operations like rotations and blurs? Read and modify individual pixels from the image? Simply save image data encoded that way to a file?

[raymanfx]

I think me and @tylerhawkes first and foremost want to convert the buffer into something we can render easily (which will be RGB888 most of the time). For YUY2 this means unpacking macropixels into YUV image pixels first and then converting those to RGB (which is what my crate is designed to do right now).

[tylerhawkes]

Yeah, @raymanfx is correct that I am getting a YUV buffer from a kernel driver and I need to convert it to rgb before I do anything else with it. That's really that only thing that I would need to drop my opencv dependency since I can blur and resize rgb images.

[Tecol87]

I also need it to convert YUYV to RGB. Most cheap v4l2 devices support YUYV, while not supporting anything else.

[raymanfx]

I also need it to convert YUYV to RGB. Most cheap v4l2 devices support YUYV, while not supporting anything else.

Here's an example of doing that using ffimage v0.10.0 from crates.io : https://github.com/raymanfx/ffimage/blob/next/ffimage-yuv/benches/convert.rs#L29. Disclaimer: I'm the author of that crate.