openvino icon indicating copy to clipboard operation
openvino copied to clipboard

[GPU] Baseline InferRequest changes for dynamic shapes support

Open andrew-k-park opened this issue 3 years ago • 1 comments

Details:

  • Divide InferRequest into InferRequest with API 2.0 and InferRequest with legacy
  • Remove cldnn::layout depedency from create_device_blob
  • Add reinterpret_device_blob and ov::PartialShape constructor in layout temporarily

andrew-k-park avatar Aug 03 '22 17:08 andrew-k-park

Lets check this with networks benchmark

yeonbok avatar Aug 04 '22 05:08 yeonbok

/azp run openvino-win

andrew-k-park avatar Aug 16 '22 15:08 andrew-k-park

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar Aug 16 '22 15:08 azure-pipelines[bot]

I still see some legacy stuff in new request impl and looks like handling of dynamic shapes is missing as well. Example from GetBlob method:

        if (isDynamic) {
            // code below is related to legacy dyn batch impl. Not handling for new dyn shapes flow.
            if (m_graph->GetMaxDynamicBatchSize() > 1) {
                SizeVector outDims = data->getTensorDesc().getDims();
                outDims[m_graph->GetOutputDynBatchDims()[name]] = m_curBatch;
                data->getTensorDesc().setDims(outDims);
            }
        } else {
            checkOutputBlob(data, name, foundOutput);
        }

Please review each method carefully and ensure that legacy features handling is removed from new impl and dynamic shapes processing is added.

Cleaned up the remaining legacy feature handling. Please review again to see if there is any dynamic shape processing that needs to be added.

andrew-k-park avatar Aug 17 '22 13:08 andrew-k-park

@andrew-k-park Could you try to add some simple functional test for dynamic case and check infer request behavior in it? Maybe activation primitive will work...

vladimir-paramuzov avatar Aug 17 '22 13:08 vladimir-paramuzov