Dynamic Shape models forbidden in DAG Scheduler?

[luvwinnie]

I have a feature extractor with input shape (-1,-1,-1,-1) which refers to (b,c,h,w) and output of (-1,-1,512) refers to (b,num_steps,512). I write a config file with following model_config_list and try to make a simple pipeline. However it seems like the DAG Scheduler not allow to use dynamic parameters? Can anyone help me with this?

OS: Ubuntu 20.04LTS Docker Container: openvino/model_server:latest

{
    "model_config_list": [
        {
            "config": {
                "name": "feature_extractor",
                "base_path": "/opt/ml/models/feature_extractor",
                "shape": "auto"
            },
            "plugin_config": {
                "CPU_THROUGHPUT_STREAMS": "CPU_THROUGHPUT_AUTO"
            }
        }
    ],
    "pipeline_config_list": [
        {
            "name": "general_model",
            "inputs": [
                "inputs"
            ],
            "nodes": [
                {
                    "name": "feature_extractor",
                    "model_name": "feature_extractor",
                    "type": "DL model",
                    "inputs": [
                        {
                            "input": {
                                "node_name": "request",
                                "data_item": "inputs"
                            }
                        }
                    ],
                    "outputs": [
                        {
                            "data_item": "cnn_features",
                            "alias": "cnn_output"
                        }
                    ]
                }
            ],
            "outputs": [
                {
                    "label": {
                        "node_name": "feature_extractor",
                        "data_item": "cnn_output"
                    }
                }
            ]
        }
    ]
}

[2022-08-08 07:36:31.675][1][modelmanager][info][modelmanager.cpp:110] Available devices for Open VINO: CPU, GNA
[2022-08-08 07:36:31.679][1][serving][info][model.cpp:38] Getting model from /opt/ml/models/feature_extractor
[2022-08-08 07:36:31.679][1][serving][info][model.cpp:45] Model downloaded to /opt/ml/models/feature_extractor
[2022-08-08 07:36:31.679][1][serving][info][model.cpp:145] Will add model: feature_extractor; version: 1 ...
[2022-08-08 07:36:31.679][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model feature_extractor status change. New status: ( "state": "START", "error_code": "OK" )
[2022-08-08 07:36:31.679][1][serving][info][modelinstance.cpp:870] Loading model: feature_extractor, version: 1, from path: /opt/ml/models/feature_extractor/1, with target device: CPU ...
[2022-08-08 07:36:31.679][1][serving][info][modelinstance.cpp:875] Some inputs shapes for model feature_extractor are set to auto
[2022-08-08 07:36:31.679][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model feature_extractor status change. New status: ( "state": "START", "error_code": "OK" )
[2022-08-08 07:36:31.679][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model feature_extractor status change. New status: ( "state": "LOADING", "error_code": "OK" )
[2022-08-08 07:36:31.840][1][modelmanager][info][modelinstance.cpp:414] Input name: inputs; mapping_name: inputs; shape: (-1,-1,-1,-1); precision: FP32; layout: N...
[2022-08-08 07:36:31.840][1][modelmanager][info][modelinstance.cpp:465] Output name: cnn_features; mapping_name: cnn_features; shape: (-1,-1,512); precision: FP32; layout: N...
[2022-08-08 07:36:31.960][1][modelmanager][info][modelinstance.cpp:688] Plugin config for device: CPU
[2022-08-08 07:36:31.960][1][modelmanager][info][modelinstance.cpp:692] OVMS set plugin settings key: CPU_THROUGHPUT_STREAMS; value: CPU_THROUGHPUT_AUTO;
[2022-08-08 07:36:31.961][1][serving][info][modelinstance.cpp:771] Loaded model feature_extractor; version: 1; batch size: -1; No of InferRequests: 5
[2022-08-08 07:36:31.961][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model feature_extractor status change. New status: ( "state": "AVAILABLE", "error_code": "OK" )
[2022-08-08 07:36:31.961][1][serving][info][model.cpp:85] Updating default version for model: feature_extractor, from: 0
[2022-08-08 07:36:31.961][1][serving][info][model.cpp:95] Updated default version for model: feature_extractor, to: 1
[2022-08-08 07:36:31.961][1][modelmanager][info][modelmanager.cpp:447] Configuration file doesn't have custom node libraries property.
[2022-08-08 07:36:31.961][1][modelmanager][info][modelmanager.cpp:332] Reading pipeline: general_model configuration
[2022-08-08 07:36:31.961][1][serving][info][modelchangesubscription.cpp:25] Subscription to model: feature_extractor from general_model
[2022-08-08 07:36:31.961][1][modelmanager][error][pipelinedefinition.cpp:414] Validation of pipeline: general_model definition failed. Node name: feature_extractor used model name: feature_extractor with dynamic batch/shape parameter which is forbidden.
[2022-08-08 07:36:31.961][1][modelmanager][info][pipelinedefinitionstatus.hpp:75] Pipeline: general_model state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent:
[2022-08-08 07:36:31.961][1][modelmanager][error][pipeline_factory.cpp:63] Validation of pipeline definition: general_model failed: Value of provided parameter is forbidden
[2022-08-08 07:36:31.961][63][modelmanager][info][modelmanager.cpp:832] Started model manager thread
[2022-08-08 07:36:31.961][64][modelmanager][info][modelmanager.cpp:850] Started cleaner thread```

I hope that using DAG Scheduler to inference my input to output. Currently I need to use the following method to inference which slower than directly using Pytorch on CPU.

        features = client.predict({"inputs": input_data}, "feature_extractor")
        output = client.predict({"inputs":features}, "encoder")
        preds = client.predict({"inputs":output["features"]}, "classifier")

[mzegla]

The log is a misleading. It's "auto" parameter that is forbidden, but dynamic shapes (with -1 dimensions) should work.

Could you remove "shape": "auto" from your config and try again?

[dkalinowski]

@luvwinnie any update on your issue?

[luvwinnie]

@dkalinowski Yes, I'm able to use dynamic shape without setting the shape to "auto"! Thanks a lot!