feat(ndi): add UYVY/UYVA support in NDI consumer

[krzyc]

This is proof of concept for my internal tests, so I it will need some work and optimization before merge. But I think that idea of supporting more NDI pixel formats is useful.

This commit adds PIXEL_FORMAT (pixel-format) parameter which accepts values like:

• BGRA (default),
• UYVA,
• UYVY.

What do you think?

[Julusian]

Hmm.. However this is done on the cpu will likely be a noticable cpu hog, but the mixer flow doesnt really allow for producing multiple pixel formats currently so I dont think that can be improved on easily.

First observation is that the send() function here really needs to be in its own thread, so that it doesnt block the rest of the channel ticking while this conversion is performed.

But perhaps instead of doing the conversion manually, ffmpeg could be used to do it? I would expect that to allow for easier handling of multiple formats and to perform better. The ffmpeg code in the decklink producer might be a good starting point, I expect it just needs the filter string changed. Perhaps that code could be refactored to be shared too?

[krzyc]

@Julusian, thanks for your advices.

(...) mixer flow doesnt really allow for producing multiple pixel formats currently (...)

Maybe we should allow mixer to serve frames to consumers in requested pixel format in one of future releases?

First observation is that the send() function here really needs to be in its own thread, so that it doesnt block the rest of the channel ticking while this conversion is performed.

I have changed this behavior and added marking dropped frames in diag.

But perhaps instead of doing the conversion manually, ffmpeg could be used to do it? (...) Perhaps that code could be refactored to be shared too?

It sounds interesting, but i do not know whether we should rely on ffmpeg in such basic stuff. It is a design question, so I will try to follow your advices.

I do not know if ffmpeg supports producing data to UYVY (32 bits per 2 pixels) buffer followed by separate alpha buffer (8 bits per pixel) - it does not look like standard conversion to me.

And because lack of free NDI tools and documentation I do not know what format is actually sent by NDI library. vMix shows BGRA as BGRA, UYVY as UYVY, but UYVA as BGRA. I wonder whether it is true or vMix display bug. If true - maybe whole effort could be changed in producing BGRA and BGRX and rest of conversions is done under-the-hood by NDI SDK? edit: It looks that NDI internally always uses YUV color space so conversion will be needed only for separated alpha channel (SDK: "Use UYVY or UYVA color if possible, as this avoids internal color conversions.")

I will probably also like to add one more thing - allow sending two separate streams - FILL and KEY in UYVY (?) format.

[ianshade]

What is the purpose of doing this conversion here? NDI converts to UYVY/UYVA internally, and AFAIK it uses vector instructions, so it does this as fast as a CPU can do it. It would be much better to:

• enable consumers to request frames in other pixel formats and do the conversion on the GPU for them. (I started working this and might share it soon)

and/or

• enable consumers to receive textures and do whatever they want with them, for example convert them to other pixel formats before downloading (NDI Consumer) or just display them without the need of going GPU->CPU->GPU (Screen Consumer)

[krzyc]

Conversion was made due to my misunderstanding of NDI library and will be removed.

This change will be reduced to alpha parameter - there will be options to send NDI with no-alpha, internal alpha (current behavior) and external alpha (separate stream). This will allow CCG to cooperate with software which is not compatible with alpha-enabled streams and requiring separate alpha channel stream. But I have to do some tests before pull request.

Your ideas regarding requesting different pixel formats by consumers are worth implementing.

[Julusian]

It looks like this isnt beneficial in its current form, as the ndi library will handle the conversion automatically.

If the later suggestion of supporting disabling alpha in the stream is still useful, or for sending the alpha as a separate stream, a new PR will be welcome