Slow ONNX inference

[airogachev]

After the "default" model was converted into ONNX format its inference speed on GPU decreased nearly 4x times. Are there any solution or updates related to onnx inference?

[Adamage]

I dont really know but as a hobbyist I can suggest to check if the code maybe copies/creates again the ONNX graph multiple times, and maybe that can be alleviated somehow. Maybe you can reuse the ONNX operations etc. And check the default parameters, maybe you need to provide some inference params

[haobo724]

I met same issue! I used default pytorch2onnx.py script in repos. Then I got the onnx model but the inference speed is decreased (from 0.13s-0.21s).