The AMD AMF encoder works as expected with 2 b-frames; stutters/choppy output with 3 b-frames

[cs9kc]

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

Git

OBS Studio Version (Other)

Master branch; build 27.2.0-f6f6690cc-Win-x64

OBS Studio Log URL

N/A

OBS Studio Crash Log URL

No response

Expected Behavior

Expected behaviour using H264 with the new AMD AMF encoder from the former obs-amf branch are as follows... when using the following settings, output is expected to be CBR, 7Mbit bitrate, with a small number of ancillary parameters set, and most importantly, H264's new B-frames set to 2. The resulting video is as expected, 60fps 1080p with no dropped frames and a constrained bitrate, ideal for streaming. Results were the same with BReferenceEnable=false as well.

RateControlMethod=1 Profile=100 ProfileLevel=42 QualityPreset=2 IDRPeriod=120 TargetBitrate=7000000 VBVBufferSize=14000000 MaxNumRefFrames=4 FillerDataEnable=true EnforceHRD=true DeBlockingFilter=true HighMotionQualityBoostEnable=false EnableVBAQ=false MaxConsecutiveBPictures=2 BPicturesPattern=2 BReferenceEnable=true

These settings achieve the above stated goal.

Current Behavior

The observed behaviour when the exact same settings are used, but B-frames are changed to 3, results in regular encoder overloads with stuttery, inconsistent playback. Results were the same with BReferenceEnable=false as well.

RateControlMethod=1 Profile=100 ProfileLevel=42 QualityPreset=2 IDRPeriod=120 TargetBitrate=7000000 VBVBufferSize=14000000 MaxNumRefFrames=4 FillerDataEnable=true EnforceHRD=true DeBlockingFilter=true HighMotionQualityBoostEnable=false EnableVBAQ=false MaxConsecutiveBPictures=3 BPicturesPattern=3 BReferenceEnable=true

Steps to Reproduce

1. Use the OBS Studio build 27.2.0-f6f6690cc
2. Record a 1080p60 clip using the settings under "Expected Behaviour" using an RDNA2 GPU
3. Record a 1080p60 clip using the settings under "Current Behaviour" using an RDNA2 GPU
4. Compare the two, both visually as you watch the clips, and in an analysis program such as Elecard StreamEye

My theory is that, because 2 b-frames work fine, and 3 b-frames don't, and the encoder usage never goes above 80%, yet the encoder says that it is overloaded when using 3 b-frames, that something in OBS Studio's AMF implementation isn't currently set up in way that is compatible with 3 b-frames.

Anything else we should know?

Thank you all again for your hard work, especially @jp9000. Logs can be provided upon request, though I will only do so privately.

Windows 11 22H1, 5800X, RX 6900 XT

Related to https://github.com/GPUOpen-LibrariesAndSDKs/AMF/issues/325

[cs9kc]

The mystery deepens:

I have found a configuration that works properly with 3 B-frames (including with all combinations of BReferenceEnable and RateControlSkipFrameEnable).

Enabling Preanalysis Component allows 3 B-frames to function properly; example below:

RateControlMethod=1 Profile=100 ProfileLevel=42 QualityPreset=2 IDRPeriod=120 TargetBitrate=7000000 MaxNumRefFrames=4 HalfPixel=true QuarterPixel=true FillerDataEnable=true EnforceHRD=true DeBlockingFilter=true HighMotionQualityBoostEnable=false EnableVBAQ=false RateControlPreanalysisEnable=true MaxConsecutiveBPictures=3 BPicturesPattern=3 BReferenceEnable=false EnablePreAnalysis=true AdaptiveMiniGOP=false PASceneChangeDetectionEnable=false

AdaptiveMiniGOP is still broken, but I have not yet determined if that is an OBS Studio thing or an AMF thing.

[cs9kc]

Even weirder, it seems to be that 3 B-frames only works without dropping frames, when the PreAnalysis Component is enabled, and PATemporalAQMode=1 is set.

I'm not sure why.

[lextra2]

@cs9kc off-topic, but have you tried PALongTermReferenceEnable=true ? If it works anything like this it should be a decent quality improvement

[cs9kc]

@lextra2 I have tried it with HEVC encoding and it does indeed help, since HEVC doesn't have B-frames. PE+PA+(other PA options) seem to be the best of all worlds with HEVC encoding right now, and I'm okay with that.

That is to say, B-frames and LTR frames can't be used together in H264 encoding.

[lextra2]

That is to say, B-frames and LTR frames can't be used together in H264 encoding.

Oh, good to know. Thanks!

[cs9kc]

Fixed in https://github.com/obsproject/obs-studio/commit/07df6548fdb88e093eb0901106a882ad73ad9848

Thanks, Jim! <3