Decreasing inference time on cpu

[M-Zubair10]

trafficstars

Thanks for this awesome model, it does evaluate good with pretrained model

Right now i am getting 15s average inference time, any way to reduce it to 2-3s

[SlongLiu]

It is a good question, but we have not explored it yet. The straightest way is to deploy the model like ONNX.

[GeorgePearse]

@SlongLiu should the operations in the model already support ONNX export? e.g.

code vaguely similar to the below should work?

# Export the model
torch.onnx.export](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export)(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "super_resolution.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                'output' : {0 : 'batch_size'}})
                                

I'll look in to optimising it with tools like OpenVINO if it does.

[Weizhongjin]

i try to use torch.onnx.export to transfer grounding-dino to onnx, it seems like some problem , such as several logical operators are not supported by onnx. for example: torch.onnx.symbolic_registry.UnsupportedOperatorError: Exporting the operator ::_ior to ONNX opset version 13 is not supported do you know which part of model make this problem. @GeorgePearse

[oylz]

here

[Dratlan]

i try to use torch.onnx.export to transfer grounding-dino to onnx, it seems like some problem , such as several logical operators are not supported by onnx. for example: torch.onnx.symbolic_registry.UnsupportedOperatorError: Exporting the operator ::_ior to ONNX opset version 13 is not supported do you know which part of model make this problem. @GeorgePearse

Have you solve it please

[kobic8]

@SlongLiu should the operations in the model already support ONNX export? e.g.

code vaguely similar to the below should work?

# Export the model
torch.onnx.export](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export)(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "super_resolution.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                'output' : {0 : 'batch_size'}})
                                

I'll look in to optimising it with tools like OpenVINO if it does.

am also interested in onx export code of GDINO