AMF
AMF copied to clipboard
GPUOpen-LibrariesAndSDKs
MJPEG video decoder initialized in D3D11 and YUY2 mode produces NV12 textures
As the title says, AMFVideoDecoderUVD_MJPEG with InitDX11 & Init AMF_SURFACE_YUY2 initialization outputs DXGI_FORMAT_NV12 textures. I believe it is expected to run internal converter and deliver DXGI_FORMAT_YUY2 instead.
Or if this is expected behavior, I beleive the component should not advertise support for "non-native" YUY2 at all.
It might also need some additional comment in what sense NV12 is "native" here. It's definitely preferable in terms of compatibility of sorts, but JPEGs are quite so often 4:2:2. Does this mean that AMD HW decoder produces "oversubsampled" NV12 output and then converts to YUY2 as requested? Or initialization in YUY2 mode results in 4:2:2 output without intermediate NV12 stage.
There are two issues here:
- I see a problem in the implementation. It forces NV12 for MJPEG - I will investigate why it was done and correct if possible.
- Unfortunately in MJPEG it is unknown till the first frame decoded if the stream is 4:2:0 or 4:2:2. At the same time the HW decoder should allocate outputs in advance. So if in the end formats don't match, we are forced to do an internal conversion to the allocated (format from Init()).
In general, I would expect the component to also handle transparently 420/422 changes from frame to frame as well. However, the point here is that if AMF component is designed to take specific fixed surface format during initialization, then it is assumed that it can handle any necessary conversions automatically.
I don't know the details of the hardware implementation and I have not checked the subsampling of the signal I used for my checking but changes are highest that it's 422 and so component already does some sort of conversion to NV12, either at hardware or software level. Given all this, I assume the problem is rather at AMF component implementation level lacking accurate code paths to match input signal subsamplnig with the requested format.
Yes, I've fixed that. Now AMF decoder will respect the output format client app asked for MJPEG, similar to other codecs. If it doesn't match the actual stream - the internal conversion will kick-in. The fix should come in one of the next hotfix drivers. The dynamic change would be a big new feature and would require HW reinitialization and delays in playback. It should be not too many streams with variable UV sampling resolution.
Out of curiosity, will this fix propagate to Media Foundation Transform (C:\Program Files\Common Files\ATI Technologies\Multimedia\amf-mft-mjpeg-decoder*.dll)? To my best knowledge AMF runtime is statically linked there, and probably not the latest snapshot of it.
The fix will be common for AMF and the MJPEG MFT. Both DLLs come with one driver package. The only difference is that MFT supports NV12 and YUY2 outputs (with optional conversion if needed) only while AMF can do more formats.
Last time I checked MFT exhibited a different colorspace issue. Opposite to the problem reported here, MFT failed to support NV12 offering only YUY2. FYI also shows a bunch of other issues, such as creating a thread per processed frame.
Just retested, WMP and Movies&TV use this MFT - don't see threading issue. The MFT supports both YUY2 and NV12 connections.
To be honest I lost my interest after the threading issue. With a high frame rate camera 640x360@260 the MFT is unable to process feed in real time just because of threading. Other observations:
- NV12 output is indicated in registration but does not work - cannot be configured; together with AMF issue it appears that the MFT is forced into implicit conversion
- debug output prints some AMF message about draining, apparently the MFT falls into some unexpected condition
- MFT registration does not have a flag that indicates the MFT is async, which it apparently is
- MFT ignores D3D11 BIND* flags for textures passed by caller
- threads are created and destroyed at the rate of incoming frames
Maybe there is something else, but threading is a show stopper.
I mounted it for a live camera and Source Reader API. I am guessing that file playback might be fine (after all I did see proper YUY2 output apart of performance impact). I am pretty sure that MFT itself is broken because putting Microsoft's software or a non-AMD wrapper over AMF immediately resolves the issues.
Also FYI MFT form factor could offer means to re-negotiate the texture format depending on actual 444/422/420 type of JPEG. That is, presumably AMF+MFT could offer a low overhead combination to never convert internally and leave it to the app.
However since MFT is a wrapper over AMF it looks unrealistic to avoid conversions: internally AMF is fixed to certain format, on first frame there might be AMF conversion enabled, then MFT might have another conversion on top of that. And finally, AMF might trigger an error condition is JPEG subsampling changes. Maybe it's something you would want to put on your low priority suggestion list.
And I am not sure what is the technical problem about getting notified on output availability, but maybe AMF could have notified that output is ready? It is not exactly specific to MJPEG, perhaps even more important with encoding.
few things:
- NV12 - correct we need to fix it.
- Registration: according to MSDN, hardware MFTs don't have to set MFT_ENUM_FLAG_ASYNCMFT flag: https://docs.microsoft.com/en-us/windows/win32/api/mfapi/nf-mfapi-mftregister, It should have MF_TRANSFORM_ASYNC attibute.
- This MFT creates a single polling thread. Could you please grab the stack for these threads and check from which DLL it originlates?
- Bind flag - good point, opened a ticket.
- Did you try Windows Camera app for your camera?
- Polling vs callback for encoding in AMF - I know the that it is a pain, stay tuned.
- Out of curiosity: what is the real use case when MJPEG stream changes from 4:2:0 to 4:2:2 in real time?
2 - MS documentation is a bit confusing here, after re-reading it I think your point is correct and it's another vendor who does MFT_ENUM_FLAG_ASYNCMFT | MFT_ENUM_FLAG_HARDWARE is not doing it right
3 - I will try to re-create this on the weekend
5 - the camera is absolutely fine, works with Camera app, works with DirectShow. However it offers both NV12 and MJPG and so Camera app might be just bypassing MJPG; as I mentioned software and non-AMD MFT cope with decoding smoothly
7 - it's an academic question ATM, however I have never yet met a mention that subsampling is expected to be fixed. It's quite the opposite - each individual frame i decodable no matter what and decoders do not normally apply restrictions. Some time in past I used to see a variant of camera, esp. such as dual lens and day/night that switch between modes following light conditions. As far as I remember some could have subsampling changed with those switches and this happened within a continuous stream. Even though I have such example, this is still an edge case. However your component is general purpose and so I think it might catch another real world example
Some more details on MFT problem:
A. I mentioned a message related to draining, here it goes:
2020-01-15 19:21:41.501 54C8 [AMFAsyncMFTBase] Error: ........\private\impl\mft\mft-framework\AsyncMFTBase.cpp(1455):AsyncMFTBase(0) MFT Polling Thread: QueryOutput() returned AMF_EOF, but MFT is not draining!
It is emitted just once at the beginning of processing.
B. Snapshot shows your polling thread coming from amf-mft-mjpeg-decoder64.dll, it's not going away
C. It is a sort of hard to capture which threads are coming and going because it looks like they are workers of some API, that is not directly yours and not directly mine.
Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
Visual Studio is just filling debug output with notifications that threads keep exiting. I'd guess you are using some "heavy" API calls that create a worker. Or who knows maybe it's related to the pixel format conversion the MFT does.
D. The problem is unrelated to camera. I hooked Logitech C930e webcam and with some its 1920x1080@20 MJPG and the behavior is exactly the same.
MF topology in question is straightforward:
{ WebCam -> your MJPEG MFT -> } -> SourceReader API
Nothing else, no converters, no third party components.
A. this should be ignored. B. Yes this thread is always running but only one per decoder instance. C. The threads can only run from Media Foundation. We have no control. D. I have C920. It can output MJPEG. You can send me a sample app - binary or source code. You can also record mftrace log to see what is going on. Or you can modify "CaptureVideo" app from AMF samples to force AMD MJPEG decoder in the MediaSource. By default, it uses MSFT MFT for decoding. To run it easily, uncheck everything in "Options" menu before clicking on "Run".
C - well, you have no direct control but you are likely trigger this activity to the state thee decoder performance is deteriorated.
From what I remember SDK MF samples could be unable to reproduce this problem as none of them is using Source Reader. Standard samples are Media Session based and there might be differences in MFT management.
To be honest CaptureVideo has about 100x code compared to what is relevant to this problem.
Nevertheless I think I see something close to my original symptom, I will push a branch with my edit.
There is a lot of converter errors whcih are probably related to one of the two YUY2/NV12 bugs we mentioned above.
Between them however there are also lots of threading messages.
The thread 0x54bc has exited with code 0 (0x0).
The thread 0x78c0 has exited with code 0 (0x0).
2020-01-15 23:14:55.731 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x5df8 has exited with code 0 (0x0).
2020-01-15 23:14:55.768 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x638c has exited with code 0 (0x0).
The thread 0x20f4 has exited with code 0 (0x0).
2020-01-15 23:14:55.845 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x2714 has exited with code 0 (0x0).
2020-01-15 23:14:55.884 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x56fc has exited with code 0 (0x0).
2020-01-15 23:14:55.942 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x73c8 has exited with code 0 (0x0).
The thread 0x5208 has exited with code 0 (0x0).
2020-01-15 23:14:55.980 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x4c70 has exited with code 0 (0x0).
2020-01-15 23:14:56.010 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x7620 has exited with code 0 (0x0).
The thread 0xdbc has exited with code 0 (0x0).
2020-01-15 23:14:56.067 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x5568 has exited with code 0 (0x0).
The thread 0x2cec has exited with code 0 (0x0).
2020-01-15 23:14:56.135 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
The thread 0x79ac has exited with code 0 (0x0).
2020-01-15 23:14:56.166 8624 [AMFVideoConverterImpl] Error: ..\..\..\..\..\runtime\src\components\VideoConverter\VideoConverterImpl.cpp(644):Assertion failed:Invalid input surface format YUY2. Expected NV12
It required more changes and one more change in MFT. I have it working. I do see the threads coming. I believe this is because the MFT is ASYNC - probably other MFTs are not. I think the threads should not be an issue, the problem was in format support. Send me a private email if you want to try: mikhail'dot'mironov'att'amd'dot'com
I have my wrapper over AMF async, threads are fine. Also in the app where I found it there is no AMFVideoConverterImpl error and decoder is generally operational, in terms of producing output textures, threading problem exists though.
Currently using Movavi's https://www.movavi.com/videoconverter/ to prevent threading. Works really fine for now, also looking for the outcome of this all.
@jotaro-eng don't spam with your self-promotion here. Needless to mention the product you advertise has nothing to do with the original issue (already fixed or on the way to be fixed by AMD).
