decklink: Output BGRA only when supported

[DDRBoxman]

Description

Currently if you select a mode that doesn't support RGBA on Decklink output the image is corrupted.

Fixes https://github.com/obsproject/obs-studio/issues/10380

Motivation and Context

https://github.com/obsproject/obs-studio/issues/10380

How Has This Been Tested?

Sent video to the Ultra Studio 4k mini with multiple video modes

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [x] I have included updates to all appropriate documentation.

[DDRBoxman]

I tweaked the logic in decklink-output.cpp to actually match correctly on HDR and keyer. It worked on my device since it supports both but this should fix compatibility on more devices.

[norihiro]

The commit message has a too long description in a line, 107 characters, limit 72.

Not all devices support the BGRA pixel format in all video modes. Make sure we fallback to YUV when needed.

[RytoEX]

Setting back to draft per off-thread conversation that the open review thread requires writing new shaders to resolve.

[RytoEX]

Setting back to draft per off-thread conversation that the open review thread requires writing new shaders to resolve.

I presume this is still the case?