[Bug]: AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY causes encoder initialization failure on pre-RDNA cards starting with 23.7.1 driver

[cgutman]

Describe the bug Setting AMF_VIDEO_ENCODER_USAGE to AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY on the AMFVideoEncoderVCE_AVC component causes the encoder to fail to initialize on at least Polaris GPUs (seems to be all pre-RDNA cards).

This was originally reported as a Sunshine bug in https://github.com/LizardByte/Sunshine/issues/1559 but the root cause appears to be a bug in the GPU driver or AMF runtime. As a remote gaming app, Sunshine defaults to the ultra-low-latency preset which exposes this bug.

To Reproduce Steps to reproduce the behavior:

1. Open the EncoderLatency sample project
2. Change the default usage value of AMF_VIDEO_ENCODER_USAGE_TRANSCODING to AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY in EncoderLatency.cpp. A patch to accomplish is attached below.
3. Run the EncoderLatency sample and see error (a breakpoint will also trigger)

Patch to EncoderLatency sample that exposes the bug:

diff --git a/amf/public/samples/CPPSamples/EncoderLatency/EncoderLatency.cpp b/amf/public/samples/CPPSamples/EncoderLatency/EncoderLatency.cpp
index 82e496f..48278f8 100644
--- a/amf/public/samples/CPPSamples/EncoderLatency/EncoderLatency.cpp
+++ b/amf/public/samples/CPPSamples/EncoderLatency/EncoderLatency.cpp
@@ -308,8 +308,8 @@ AMF_RESULT SetEncoderDefaults(ParametersStorage* pParams, amf::AMFComponent* enc
         AMF_RETURN_IF_FAILED(PushParamsToPropertyStorage(pParams, ParamEncoderUsage, encoder));

         // AMF_VIDEO_ENCODER_USAGE needs to be set before the rest
-        res = encoder->SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_TRANSCODING);
-        AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_TRANSCODING) failed");
+        res = encoder->SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY);
+        AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY) failed");

         // initialize command line parameters
         AMF_RETURN_IF_FAILED(PushParamsToPropertyStorage(pParams, ParamEncoderStatic, encoder));

Setup (please complete the following information):

• OS: Windows 10 22H2
• Driver Version: 23.8.1
• GPU: Radeon RX 480
• Which component has the issue: Encoder (AMFVideoEncoderVCE_AVC)

Debug Log (please upload or paste):

2023-08-23 22:27:03.972      6C4 [VCEEncoderTrace] Warning: GetProperty RateControlPreanalysisEnable failed in AMFPropertyStorageExImpl with return code:0x3
2023-08-23 22:27:03.972      6C4 [AMFEncoderVCE]   Error: ..\..\..\..\..\runtime\src\components\EncoderVCE\EncoderVCEImpl.cpp(3709):AMF_ERROR 3 : AMF_ACCESS_DENIED: GetCaps() - unable to get preencode mode
2023-08-23 22:27:03.972      6C4 [AMFEncoderVCE]   Error: ..\..\..\..\..\runtime\src\components\EncoderVCE\EncoderVCEImpl.cpp(875):AMF_ERROR 3 : AMF_ACCESS_DENIED: Init() - Failed to query EncoderCaps
2023-08-23 22:27:03.972      6C4 [EncoderLatency]   Error: C:\Users\camer\Downloads\AMF-1.4.30\amf\public\samples\CPPSamples\EncoderLatency\EncoderLatency.cpp(627):AMF_ERROR 3 : AMF_ACCESS_DENIED: encoder->Init() failed

Expected behavior The encoder should initialize correctly when configured with the AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY usage value.

Screenshots N/A

Additional context The HEVC encoder does print a warning about pre-analysis, but initializes correctly:

2023-08-23 22:25:57.204     3604 [AMFEncoderHEVC] Warning: SetProperty HevcEnablePreAnalysis not found

It looks like some missing/incorrect error handling in the AMFVideoEncoderVCE_AVC encoder. Hopefully it's a simple fix.

[cgutman]

I bisected the bug on my RX 480 system and found: Last working driver: 23.5.2 First broken driver: 23.7.1

So the regression was introduced in the 23.7.1 driver. Based on user reports, this issue also affects RX Vega 64.

It can also be reproduced with the following FFmpeg command:

ffmpeg -i <input file>.mp4 -codec:v h264_amf -usage ultralowlatency out.mp4

[cgutman]

Are there any plans to fix this regression in a future driver update?

We're still seeing a number of reports from AMD users impacted by this regression in https://github.com/LizardByte/Sunshine/issues/1559 and https://github.com/LizardByte/Sunshine/issues/1860

[rhutsAMD]

One way to work around this would be:

1. Set AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY
2. Get AMF_VIDEO_ENCODER_PREENCODE_ENABLE and check for errors
3. If step 2 failed, then set AMF_VIDEO_ENCODER_USAGE_TRANSCODING