Need support for video wall concept with display outputs from multiple threads

[APrashant]

Hi,

We are planning to composite multiple decoder output in single display window. Where, each and every decoding happening in separate thread. we found that it cannot handle it with above approach since, the SafetySurfaceBuffer and drm display handle is different for each threads.

Since we need N-N pipeline for our project as described in https://github.com/Intel-Media-SDK/MediaSDK/issues/2854. We tried to achieve this N(1-N) pipeline by creating N threads for each and every 1-N par file (without display). With this it worked as expected.

But with this multiple thread approach for N(1-N) pipeline, we couldn't add display pipeline as mentioned in the following par file. Because, composition and single display is not possible with the existing code.

For Example par file 1: (par file parsing and running in thread 1) -i::h264 1min.h264 -join -hw -async 1 -o::sink -vpp_comp_dst_x 0 -vpp_comp_dst_y 0 -vpp_comp_dst_w 320 -vpp_comp_dst_h 240 -ext_allocator -vpp_comp_only 1 -w 1920 -h 1080 -async 1 -join -hw -i::source -ext_allocator -ec::rgb4 -rdrm-DisplayPort -fps 30

par file 2: (par file parsing and running in thread 2) -i::h264 1min.h264 -join -hw -async 1 -o::sink -vpp_comp_dst_x 320 -vpp_comp_dst_y 0 -vpp_comp_dst_w 320 -vpp_comp_dst_h 240 -ext_allocator -vpp_comp_only 1 -w 1920 -h 1080 -async 1 -join -hw -i::source -ext_allocator -ec::rgb4 -rdrm-DisplayPort -fps 30

Whether it is possible to achieve N-1 display with multiple threads. i.e. each and every thread will have a decoded output - all together we need to composite in single display based on the co-ordinates of each input. If this is possible can we have any inputs that will help us in our implementation.

Thanks in advance.

[dmitryermilov]

As said in #2843, sample_multi_transcode is not a good basis. Better to write own app. You just need N instances of kind of "decoder" class which produce surfaces at output and puts them into compositor component shared between them. You either you need 1) a master component which starts all decoders, takes (in fact waits) one surfaces from each and then call compositor class or 2) tells the decoder classes to put their output directly to compositor class, which will trigger composition when it receives all N input surfaces.

[dmitryermilov]

BTW, you may look at https://github.com/dmitryermilov/sandbox/tree/jpeg_hevc/transcode. It's an example of N transcode jpeg->hevc pipeline. It's of course simpler than what you need but at least you'll get one more reference.

[APrashant]

Hi @dmitryermilov,

In N-1 Display pipeline - we are getting display only when we use -join option (which joins all session) in par file.

In N-1 Display pipeline - when we try to render the display without join - sync operation getting failed after calling VppOneFrame() in Encode() function.

VPP SyncOperation returns MFX_ERROR_UNSUPPORTED m_pmfxSession->SyncOperation(VppExtSurface.Syncp,MSDK_WAIT_INTERVAL);

The above things we are trying in the existing sample_multi_transcode with N-1 display par file.

Whether JOIN (joining sessions) is mandatory for compositing multiple inputs in single display?

Our understanding is that following code is for handling the case when join is not used in par file:

// if session is not joined and it is not parent - synchronize if (!m_bIsJoinSession && m_pParentPipeline) { // if it is not already synchronized if (DecExtSurface.Syncp) { MFX_ITT_TASK("SyncOperation"); sts = m_pParentPipeline->m_pmfxSession->SyncOperation(DecExtSurface.Syncp, MSDK_WAIT_INTERVAL); HandlePossibleGpuHang(sts); MSDK_CHECK_ERR_NONE_STATUS(sts, MFX_ERR_ABORTED, "Encode: SyncOperation failed"); } }

As mentioned initially in this issue ticket, We are using multiple threads each thread will have a decoder and display session, but we are trying to render the display in the Nth thread by collecting all the decoded output from multiple threads.

We are also facing the same SyncOperation issue in VPP, Since we couldn't join the sessions between the threads. Please help us in resolving this issue.

Thanks in advance.

[APrashant]

Hi @dmitryermilov,

We suspect that, for composition+rendering - joining the sessions are mandatory. If so, can you suggest us any possible way of achieving composition+rendering without joining the sessions and its possibility.

We have mentioned our understandings in the above comment.

Thanks in advance