Fix full/limited range and RGB color space handling

[miabaka]

Description

Motivation and Context

It fixes #8199

How Has This Been Tested?

I recorded videos with multiple combinations of color range, color space and output methods and then verified that all ok with image and file metadata

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [ ] I have included updates to all appropriate documentation.

[flaeri]

Some testing so far. I'll be using mpv, ffmpeg and vlc for decode. For ffprobe I'm running -show_entries stream to check tagging.

yuv444: libx264 seems good, partial and full, correctly decoded and matches the range.

The following codecs break so far: huffyuv, prores_ks, cineform, ffv1 (more to test) They break on full range, as we get a mismatch (data full, decode partial. Tagging is "unknown"). I'm not sure if full range is legal for cineform yuv, and that may apply to others as well.

Unsure if this is related, or poor timing. I tried to repro, but seems to be intermittent. OBS was recording with full range ffv1. Stop recording, change to partial range, record, and exception was thrown.

 	avcodec-59.dll!00007ffb2c4021fd()	Unknown
 	avcodec-59.dll!00007ffb2c0c840f()	Unknown
 	avcodec-59.dll!00007ffb2c07d800()	Unknown
 	avcodec-59.dll!00007ffb2c07e449()	Unknown
>	obs-ffmpeg.dll!receive_video(void * param, video_data * frame) Line 812	C
 	obs.dll!default_raw_video_callback(void * param, video_data * frame) Line 1786	C
 	[Inline Frame] obs.dll!video_output_cur_frame(video_output *) Line 142	C
 	obs.dll!video_thread(void * param) Line 188	C
 	w32-pthreads.dll!ptw32_threadStart(void * vthreadParms) Line 225	C
 	[External Code]	

I was in the process of testing QuickTime RLE (full range this time), at which point a very similar exception as above threw. I'm not so sure its a coincience anymore :(

 	swscale-6.dll!00007ffb877f869c()	Unknown
 	swscale-6.dll!00007ffb87833e3e()	Unknown
 	swscale-6.dll!00007ffb8781b462()	Unknown
 	swscale-6.dll!00007ffb8781c30b()	Unknown
>	obs-ffmpeg.dll!receive_video(void * param, video_data * frame) Line 786	C
 	obs.dll!default_raw_video_callback(void * param, video_data * frame) Line 1786	C
 	[Inline Frame] obs.dll!video_output_cur_frame(video_output *) Line 142	C
 	obs.dll!video_thread(void * param) Line 188	C
 	w32-pthreads.dll!ptw32_threadStart(void * vthreadParms) Line 225	C
 	[External Code]	

I stopped testing more codecs at this point.

[RytoEX]

This will not be merged if the test feedback above is not addressed.

[flaeri]

It turns out that the crashes were not related to this PR. After spending a long time testing both release and master, I would get the same crashes there. I was wrong, and I should have tested more on release prior. I genuinly had never seen those crashes before on release, and I've used both codecs several times before. I wish to apologize for that.