[Bug]: VBAQ causes severe blur on I-frames during motion in AMF encoder

[Ryu0833]

Describe the bug With my testing of the encoder (visual test), it feels like AMF_VIDEO_ENCODER_ENABLE_VBAQ has a problem with I-frames. When there is motion in the video and an I-frame is inserted, the entire frame becomes blurred. So either VBAQ cannot properly handle I-frames (causing the frame to blur completely), or there is a problem with the VBAQ implementation.

In my test, I changed the keyframe interval from 2s When the image is static, there is no problem (just a little blur, barely noticeable). But when moving, at the moment an I-frame is inserted, the image becomes very blurry.

To Reproduce Steps to reproduce the behavior using obs:

1. Set the following properties 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=false HighMotionQualityBoostEnable=true ' when test the VBAQ false
2. start the encoding with resolution=1920*1080 bitrates=8000kbps the test.mkv (the test.mkv has fix QP=16 for high quality )
3. test with VBAQ following properties 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=true HighMotionQualityBoostEnable=true '
4. start the encoding resolution=1920*1080 bitrates=8000kbps test.mkv

Setup:

• OS: Windows 11 24h2
• Driver Version 25.9.1
• GPU RX 5500 XT
• Which component has the issue Encoder

Expected behavior When VBAQ is enabled, I-frames should remain clear (similar to when VBAQ is disabled), and the entire frame should not become blurred.

Additional context This problem is consistent at both high and low bitrates — the blur issue always occurs.

[MikhailAMD]

Can you share command line for TranscodeHW sample or for FFmpeg to repro together with source file and both outputs? We need to see whole set of parameters.

[Ryu0833]

i did use obs last master and with: 1)testing with VBAQ=true 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=false HighMotionQualityBoostEnable=true ' 2)testing with VBAQ=true 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=true HighMotionQualityBoostEnable=true '

but i cant send the videos of the test with the result test

[Ryu0833]

I forgot to mention that all the tests, both with VBAQ=true and VBAQ=false, were done using CBR.

[MikhailAMD]

I've asked rate control gurus to check.

[StevenXiao-AMD]

I don’t have the board GPU RX 5500 XT used in this ticket.

Instead, I tried to use the latest AMD RX 9070 XT to reproduce the issue in latest OBS, but I can't observe the blur in image. I set exactly the same encoding parameters described in this ticket.

In fact, no matter VBAQ true or false, the quality of I-frame is very closed to its neighbor P-frame or B-frame. All frame types are good quality.

@Ryu0833 Could you please share your bitstreams, or give more context about your tests?

[Ryu0833]

I don’t have the board GPU RX 5500 XT used in this ticket.

Instead, I tried to use the latest AMD RX 9070 XT to reproduce the issue in latest OBS, but I can't observe the blur in image. I set exactly the same encoding parameters described in this ticket.

In fact, no matter VBAQ true or false, the quality of I-frame is very closed to its neighbor P-frame or B-frame. All frame types are good quality.

@Ryu0833 Could you please share your bitstreams, or give more context about your tests?

Hello, the RX 9070 XT has the new AMF encoder (VCN 5.0) with new features. I’m talking about the older versions VCN 2.0, 3.0, and 4.0. where enabling VBAQ makes the blurry frames more noticeable.

The test was done by encoding a video using a constant quantization parameter (CQP) of 16. Then, the same video was recorded twice ,once with VBAQ enabled (true) and once with VBAQ disabled (false) to compare the quality. The resolution was 1920×1080p60, with a bitrate of 8000 kbps and a keyframe interval of 2 seconds. The blur is more noticeable when VBAQ is enabled.

Since VCN 5.0 is more powerful than the previous versions, it’s possible that the VBAQ implementation maybe most be changed and behaves differently across encoder generations.

[Feng-AMD]

@Ryu0833 : I have tested using your settings and command line, with RX 5500 XT. Unfortunately I could not reproduce the issue you have mentioned.

In order for us to investigate further, it is necessary that we have your test clips, including the one you coded as source with CQP of 16.

By the way, can you please confirm that you have tested the older versions VCN 2.0, 3.0, and 4.0 and they all exhibit the same issue? Also, what is your OBS version ?

[Ryu0833]

How can I attach the test clips in a comment? Is there a way?

I’ve uploaded the test clips to Google Drive for reference.https://drive.google.com/file/d/10F3K4XEk55_kCa_6e6IHDacC4nNYCp_e/view?usp=sharing in the zip file there is : all the video encode amd hw h.264 and preset=quality profile=high test.mp4(the coded as source with CQP=16) VBAQ=true.mp4 VBAQ=false.mp4 I tried to capture the same frame to better demonstrate the blur. VBAQ=false: VBAQ=true:

I also want to clarify that I didn’t test VCN 3.0 or VCN 4.0, and I just want to show that the RX 9070 XT is not the same as the earlier generations, which are closer to each other in terms of quality. The OBS version used is 32.0.0.