Support of gaze-estimation-adas-0002 model

[sicoyle]

This is my err msg: ERROR: from element /GstPipeline:pipeline0/GstGvaClassify:gvaclassify5: base_inference plugin intitialization failed Additional debug info: /root/gst-video-analytics/gst/inference_elements/base/inference_singleton.cpp(136): acquire_inference_instance (): /GstPipeline:pipeline0/GstGvaClassify:gvaclassify5: Failed to construct OpenVINOImageInference Config for layer precision does not contain precision info for layer: head_pose_angles ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ... Freeing pipeline ...

This is my Gstreamer pipeline: gst-launch-1.0 --verbose filesrc location=someVideo.mp4 ! decodebin ! videoconvert ! \ gvadetect model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/intel/person-detection-0200/FP32/person-detection-0200.xml threshold=0.5 ! gvatrack tracking-type=short-term ! gvadetect model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-retail-0004/FP32/face-detection-retail-0004.xml model-proc=/opt/intel/openvino_2021/data_processing/dl_streamer/samples/model_proc/face-detection-retail-0004.json ! queue ! \ gvaclassify model=/opt/intel/openvino_2021/deployment_tools/tools/model_downloader/intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml model-proc=/opt/intel/openvino_2021/data_processing/dl_streamer/samples/gst_launch/face_detection_and_classification/model_proc/age-gender-recognition-retail-0013.json object-class=face ! queue ! \ gvaclassify model=/opt/intel/openvino_2021/deployment_tools/tools/model_downloader/intel/emotions-recognition-retail-0003/FP32/emotions-recognition-retail-0003.xml model-proc=/opt/intel/openvino_2021/data_processing/dl_streamer/samples/gst_launch/face_detection_and_classification/model_proc/emotions-recognition-retail-0003.json object-class=face ! queue ! \ gvaclassify model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml ! queue ! \ gvaclassify model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/intel/facial-landmarks-35-adas-0002/FP32/facial-landmarks-35-adas-0002.xml ! queue ! \ gvaclassify model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/public/open-closed-eye-0001/FP32/open-closed-eye-0001.xml ! queue ! \ gvaclassify model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/intel/gaze-estimation-adas-0002/FP32/gaze-estimation-adas-0002.xml ! queue ! \ gvametaconvert json-indent=4 ! gvametapublish method=file file-path=/home/sam/Documents/outputResults.json ! gvawatermark ! ximagesink sync=false

I'm not sure if I am in need of a model_proc file for any of the models and this should be a feature-request and not an issue, or if this is in fact an issue with the models.

Note: I converted the open-closed-eye-0001 model from .onnx format to the openvino model format.

Related issue

[vladimir-dudnik]

you probably would like to ask this question at dlstreamer project repo

[brmarkus]

Can you try to simplify your gstreamer pipeline first, please - in order to find out which detection and/or classification, which model (s) is/are causing the problem you are seeing? That would allow to focus on the root-cause analysis.

[sicoyle]

Thank y'all for responding back so quickly.

I can confirm that if I remove the below line from the pipeline, then it works perfectly fine. gvaclassify model=/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader/intel/gaze-estimation-adas-0002/FP32/gaze-estimation-adas-0002.xml ! queue ! \

Note: I am using openvino 2021.3.394 and GStreamer 1.16.2

[adranit]

Hi @sicoyle ,

As Mark Liu commented in the issue you mentioned in the description, unfortunately gaze-estimation-adas-0002 model is not supported out-of-box. It requires special preprocessing to be written (https://github.com/openvinotoolkit/dlstreamer_gst/wiki/Custom-Processing)

"The Gaze Estimation demo is more complicated than our current design. Because it requires to calculate eye regions and pass separately left and right eye images according to gaze-estimation-adas-0002 model inputs. We don’t support such use case right now, so this pipeline won’t work.

The other requirement is, gaze-estimation-adas-0002 model has three inputs, we don’t know which one is for what data so application needs to provide ‘model-proc’ file with description of these input layouts (format, precision, etc.) as expected. But since current function doesn't support such many inputs, the pipeline still doesn't work even if model-proc file is provided."

[sicoyle]

@adranit That makes sense. Being that I am not the only one who has faced this issue, do you think it appropriate and worthwhile to create another issue (or classify this issue) to request a feature here?

Also, how are developers supposed to know (say in a case such as this one) that a pipeline may be too complicated for DL Streamer? Is that simply due to the complexity of the inputs of the gaze-estimation-adas-0002 model?

[adranit]

Hi @sicoyle , Let's rename the issue and mark it as feature request. We regularly add support for new model types from OMZ, unless of course they turn out to be a very special case.

Regarding the second question - if there is no associated model_proc file in DL Streamer repo for a model, then probably the model is not supported out-of-box. In this case I'd recommend to refer to the wiki page describing how to create model_proc file with existing pre- and post- processors. To check if a model is supported you need to use description of model's input and output layers.

The general case when the model has 2 or more input layers is not supported by DL Streamer. Gaze-estimation-adas-002 is one of the few OMZ models with more than one input, and pre-processing is not implemented for it.

[akwrobel]

This issue has been reviewed and this is not in our roadmap, and there is currently no plan to add it. We are closing this issue as a result.