[Bug]: Encoder hangs when `USAGE_HIGH_QUALITY` is used with VBR rate control on DX9 and Vulkan

[RFCArrow]

Describe the bug The encoder does not return frames when the encoder USAGE is configured to HIGH_QUALITY and the RateControlMethod is configured to VBR for the DX9 and Vulkan ENGINEs.

By contrast, the encoder does correctly return frames if any of the following are true:

• ENGINE is set to DX11
• RateControlMethod is set to QVBR
• EnabledPreAnalysis is set to false
• USAGE is set to TRANSCODING

To Reproduce

1. Build the AMF TranscodeHW sample project
2. Run the sample with either of the following sets of arguments: -INPUT input_1920x1080_8bit_420p.yuv -OUTPUT output.h264 -Usage HQ -ENGINE VULKAN -RateControlMethod VBR -INPUT input_1920x1080_8bit_420p.yuv -OUTPUT output.h264 -Usage HQ -ENGINE DX9 -RateControlMethod VBR
3. Observe the programme exits with an error code and there is no output bitstream.

Setup

• OS: Windows 11 Pro (22H2 / 22621.2283)
• Driver Version: 21.9.3
• GPU: RX 7900 XTX
• AMF: 1.4.30 (2f32635)
• Which component has the issue: Encoder

Debug Log (please upload or paste): Attached here: amf_high_quality.log

Expected behavior Encoder generates an output without errors. The output has a non-zero size and can be played back.

[rhutsAMD]

Usage set to high will default both AMF_VIDEO_ENCODER_PRE_ANALYSIS_ENABLE to true as well as AMF_PA_TAQ_MODE to AMF_PA_TAQ_MODE_2. AMF_PA_TAQ_MODE_2 only supports the encoder memory type / ENGINE set to AMF_MEMORY_DX11.

I have added an additional check for this scenario which will print a helpful message instead of failing silently. The API doc will be updated as well to include this support information for AMF_PA_TAQ_MODE_2.