obs-ffmpeg: Linux NVENC texture encoder

[derrod]

Description

WIP NVENC texture encoder for Linux, based on software encoder introduced in #10122 and OpenGL texture encoding libobs changes in #8312.

This PR really is just the last commit.

Superseedes/closes #4974

Motivation and Context

Gotta go fast.

How Has This Been Tested?

OBS Flatpak on openSUSE Tumbleweed with NVIDIA 545 drivers.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [x] I have included updates to all appropriate documentation.

[ryzendew]

Does it have av1?

[derrod]

Short AV1 clip recorded using the Flatpak build on openSUSE with the new texture sharing encoder:

https://github.com/obsproject/obs-studio/assets/3123295/e3939e68-ad72-4dca-b4d8-c4d22af0640c

[derrod]

With #8312 merged I'll take this out of draft. It works and the PR can now be reviewed independently.

[kkartaltepe]

Fallback encoders dont appear to be private and can be seen in the options, also the default encoder doesnt fallback and just fails with an error when using the wrong gpu. (couldnt actually get obs to load on the gpu atm so ill test the working path later).

[derrod]

Fallback encoders dont appear to be private and can be seen in the options, also the default encoder doesnt fallback and just fails with an error when using the wrong gpu. (couldnt actually get obs to load on the gpu atm so ill test the working path later).

Yep, I haven't implemented an automatic fallback yet so I left the non-texture encoders visible. Gotta figure out how to determine if OBS and the CUDA context run on the same GPU.

[derrod]

Updated with a commit adding a check on startup to see if mapping an OpenGL texture created by the OBS graphics context succeeds on any CUDA device. If not, only fallback encoders will be supported, otherwise the CUDA device where mapping succeeded will be used when starting a texture encoder.

This also hides the fallback encoders by default on Linux as well.

[Bleuzen]

Hi, this is really cool, used it for streaming lately, works fine :)

Only minor things noticed just now due to the addition of AV1: EDIT: separated because it is out of scope for this PR, see: https://github.com/obsproject/obs-studio/issues/10181 https://github.com/obsproject/obs-studio/discussions/10182

[tt2468]

These are all existing behaviors of the shared texture NVENC encoder and changing them is out of scope of the goals of this PR.

[Bleuzen]

Did some more testing, this time with recording. There is a noticeable change in the bitrates used. I set up OBS with and without this patch the same way and used a VLC source with the same lossless video playing for reproducible input

For example when recording NVENC H264 in Simple Mode / "High Quality, Medium File Size"...

Current OBS:

With this PR:

Same with HEVC

Before:

After:

~ 75 % increase in file size

Also in "Advanced" Output Mode with same CQP Levels

Didn't really expect that, not sure if this is supposed to happen or a bug

[tt2468]

Did some more testing, this time with recording. There is a noticeable change in the bitrates used. I set up OBS with and without this patch the same way and used a VLC source with the same lossless video playing for reproducible input

For example when recording NVENC H264 in Simple Mode / "High Quality, Medium File Size"...

Current OBS:

With this PR:

Same with HEVC

Before:

After:

~ 75 % increase in file size

Also in "Advanced" Output Mode with same CQP Levels

Didn't really expect that, not sure if this is supposed to happen or a bug

The encoder implementations are different, but nothing should have been touched in relation to the bitrate regulation. That's likely just a pre-existing difference between the ffmpeg and texture encoders.

In order to get useful results, this texture encoder PR must be tested against the current texture encoder available on Windows.

[ochilan]

@Bleuzen Could you provide details about each of the resulting files, for example using a tool like "mediainfo"? I wonder if there are unintended technical differences like e.g. chroma subsampling (4:2:0 vs. 4:4:4).

[realnc]

Applied on top of 30.1.0 and it works fine for me. I can record AV1 with NVENC and it's fast too!

Now if we only had CQ support instead of CQP, that would make OBS the perfect capture tool for image quality.

[ochilan]

The encoder implementations are different, but nothing should have been touched in relation to the bitrate regulation. That's likely just a pre-existing difference between the ffmpeg and texture encoders.

In order to get useful results, this texture encoder PR must be tested against the current texture encoder available on Windows.

I was curious and compared stock OBS 30.1 under Linux, stock 30.1 under Windows, and 30.1 + texture encoder under Linux (H.264, CQP 20, with as similar settings as possible). The resulting file sizes were:

Stock 30.1 Linux: 109.1 MiB Stock 30.1 Windows: 149 MiB Patched 30.1 Linux: 147.5 MiB

So there seems to be a difference before and after the patch under Linux, but the larger file size is actually comparable to what the Windows version already produces.

(As far as I could tell, the metadata of all videos is identical by the way, except for the video bitrates.)

[derrod]

I'm not sure if there are any differences with CQP handling in FFmpeg vs OBS, but I also don't think that's worth investigating/worrying about at this time. For now we don't want to break anything on the Windows side, but this PR is a stepping stone to a larger rework that may change things.