Clarification regarding order of data returned from MFXVideoDECODE_GetPayload()

[softworkz]

In a recent conversation, @dmitryermilov had stated this:

GetPayload returns payloads in FIFO order. Since bitstream is encodedOrder, payloads will be in encodedOrder as well. App is expected to associate payloads with frames by output timestamps "mfxU64 *ts".

Originally posted by @dmitryermilov in https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1070812574

But deeper research has brought me to the conclusion that this is not the case and that when calling GetPayload after each successful decode call, you will always get exactly those payloads which are matching the output surface from the decode call.

Pseudo-Code:

    do {
        ret = MFXVideoDECODE_DecodeFrameAsync(q->session, avpkt->size ? &bs : NULL,
                                              insurf, &outsurf, sync);
    } while (ret == MFX_WRN_DEVICE_BUSY || ret == MFX_ERR_MORE_SURFACE);

    if (*sync) {
        while (1) {
            ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload);
            // process payload and attach to outsurf
        }
    }

Please see my detailed analysis here: https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932

A look into the MSDK source code confirms this:

https://github.com/intel/MediaSDK-VPU/blob/fe578a454c0728e5b484cc93fbbd73818b7d9941/_studio/shared/umc/codec/h265_dec/src/umc_h265_task_supplier.cpp#L386-L415

This is actually a somewhat weird logic where the “isUsed” field is not about whether used or not – it’s rather a kind of counter by which the sei messages are sorted in display order.

That provides the following behavior: When you always – after a successful decode call – execute GetPayload() in a loop until no more are available, then you will automatically get the right payloads for that decoded frame.

Can you confirm this?

Thanks, softworkz