onnx-tensorrt
onnx-tensorrt copied to clipboard
onnx
Converting multi-input conv layer to tensorrt failed.
Hello guys,
I try converting a model (pytorch -> onnx -> tensorrt) with one multi-input conv layer. But it failed :( Here is the script of converting pytorch model to onnx model:
import torch
import torch.nn.functional as F
class my_conv_model(torch.nn.Module):
def __init__(self):
super(my_conv_model, self).__init__()
def forward(self, input):
kernel = torch.rand(16, 3, 1, 1)
output = F.conv2d(input, kernel, stride=1)
return output
if __name__ == "__main__":
net = my_conv_model()
input_tensor = torch.rand(1, 3, 1024, 1728)
input_names, output_names = [ "input_onnx"], [ "output_onnx" ]
torch.onnx.export(net, input_tensor, "test.onnx", verbose=True, input_names=input_names, output_names=output_names, opset_version=10)
I also use onnx python API to print some onnx model info:
output: "1"
op_type: "RandomUniform"
attribute {
name: "shape"
ints: 16
ints: 3
ints: 1
ints: 1
type: INTS
}
input: "input_onnx"
input: "1"
output: "output_onnx"
op_type: "Conv"
attribute {
name: "dilations"
ints: 1
ints: 1
type: INTS
}
attribute {
name: "group"
i: 1
type: INT
}
attribute {
name: "kernel_shape"
ints: 1
ints: 1
type: INTS
}
attribute {
name: "pads"
ints: 0
ints: 0
ints: 0
ints: 0
type: INTS
}
attribute {
name: "strides"
ints: 1
ints: 1
type: INTS
}
When I used onnx2trt tool to convert this onnx model to tensorrt engine, I got the following error:
----------------------------------------------------------------
Input filename: /root/data/Commonly_Used_Files/Model/solov2/onnx/solov2_test.onnx
ONNX IR version: 0.0.4
Opset version: 10
Producer name: pytorch
Producer version: 1.3
Domain:
Model version: 0
Doc string:
----------------------------------------------------------------
Parsing model
Building TensorRT engine, FP16 available:1
Max batch size: 32
Max workspace size: 1024 MiB
[2021-01-04 03:20:29 ERROR] _0: kernel weights has count 0 but 48 was expected
[2021-01-04 03:20:29 ERROR] _0: count of 0 weights in kernel, but kernel dimensions (1,1) with 3 input channels, 16 output channels and 1 groups were specified. Expected Weights count is 3 * 1*1 * 16 / 1 = 48
[2021-01-04 03:20:29 ERROR] Layer _0 failed validation
[2021-01-04 03:20:29 ERROR] Network validation failed.
I have read the code and it seems onnx-tensorrt actually could support multi-input conv layer. Could you help to look at this issue?
What's more, the version info is as follows:
pytorch: 1.4.0 tensorrt: 7.1.2.8 onnx-tensorrt: 7.1.0
Thanks a lot!
Best regards, Wei Zhiyi
@weizhiyi777 No, you have bug in your code, and onnx2trt currently not support multi-input conv.
See my issue you can reproduce, once you exported right onnx model, you will got error like this:
[8] Assertion failed: ctx->network()->hasExplicitPrecision() && "TensorRT only supports multi-input conv for explicit precision QAT networks!"
We only support multi-input convs for quantized networks. What is the use case of this conv?
We only support multi-input convs for quantized networks. What is the use case of this conv?
Hi @kevinch-nv multi-input convs are used in SOLOv2 network, which is a instance semantic segmentation network. One of its innovation is using features to predict weights of convs. These weights are used for subsequent conv operations. Details could be found here:
[Paper] https://arxiv.org/abs/2003.10152 [Github Repository] https://github.com/WXinlong/SOLO
I think Tensorrt could support multi-input conv ops. It would be very helpful for onnx-tenssort to support that, because more and more networks start to predict conv weight.
@weizhiyi777 Some people able convert SOLOv2 to tensorrt: https://mp.weixin.qq.com/s/gk3Rq2kmZ159gZdYNGGvMA
@jinfagang Thanks a lot! I saw this probelm was deal with by other solutions. I will try that.
I also meet this problem when I use F.conv2d. Can you tell me how to solve it? @jinfagang @weizhiyi777
@Xiaoyw1998 I didn't solve that yet but in this link maybe you could get some inspiration: https://mp.weixin.qq.com/s/gk3Rq2kmZ159gZdYNGGvMA. In this blog, the author use matrix multiplication to replace F.conv2d, which could be supported by onnx. I think you could try this solution if you have time.
@Xiaoyw1998 I didn't solve that yet but in this link maybe you could get some inspiration: https://mp.weixin.qq.com/s/gk3Rq2kmZ159gZdYNGGvMA. In this blog, the author use matrix multiplication to replace F.conv2d, which could be supported by onnx. I think you could try this solution if you have time.
Thank you for your reply. But I want to use 7x7 conv, it cannot be replaced by matrix multiplication.
@weizhiyi777 @jinfagang @Xiaoyw1998 Have you solved the problem with Multi Input Convolution?
@weizhiyi777 @jinfagang @Xiaoyw1998 Have you solved the problem with Multi Input Convolution?
No
This is still a known limitation inside TensorRT. We are planning to support this in a future release of TensorRT, in the meantime it's recommended to export your models with static conv weights if possible.
in my situation, using torch.nn.conv2d instead of F.conv2d . It seems that when using nn.conv2d, you need to initialize in_channels and out_channels，the onnxparser knows how to deal with it. But F.conv2d gives the kernel weights by hands.
in my situation, using torch.nn.conv2d instead of F.conv2d . It seems that when using nn.conv2d, you need to initialize in_channels and out_channels，the onnxparser knows how to deal with it. But F.conv2d gives the kernel weights by hands.
cause data input and kernel wieghts are two inputs
Hello @kevinch-nv. First of all, thank you very much for contributing to the open-source community!
Quick question for you: is there a branch we could pull to use this feature for "early" access instead of waiting for the next release? If not, when do you expect the next release to ship?
+1 problem with convolution dynamic weights. I think write a custom operator in tensorrt plugin may solve this problem.
@erfaneshrati @rocco-haro @wuyunnben Thank you for your patient. The release will happen in one month.
+1 waiting for dynamic weight convolution to accelerate inference time ^-^
Guys, thank you for your patient! Please check the latest TensorRT 8.5.1 release. You can find the header description here: https://github.com/NVIDIA/TensorRT/blob/main/include/NvInfer.h#L1442
Closing this issue since this feature has been released. Feel free to reopen if any further questions. Thanks!
@lansfair If you are looking for convert from PyTorch to TensorRT directly, you might try https://github.com/pytorch/TensorRT; otherwise you can export PyTorch to ONNX and let TensorRT loads the ONNX model.