TensorRT
TensorRT copied to clipboard
NVIDIA
Failure due to batch size of TensorRT 8.6.1 when running inference on NVIDIA RTX A2000 8GB Laptop GPU
Description
I am trying to convert sligtly modified version of YOSO from pytorch to TRT. I cannot make it work with batch size 8.
Can you please point me to exact step in my description which I did incorrectly? Thanks a lot in advance.
Environment
TensorRT Version: TensorRT 8.6.1
NVIDIA GPU: NVIDIA RTX A2000 8GB Laptop GPU
NVIDIA Driver Version: 535.154.05
CUDA Version: 12.2
CUDNN Version:
Operating System:
Python Version (if applicable): 3.10.12
Tensorflow Version (if applicable): N/A
PyTorch Version (if applicable): 2.2.0
Baremetal or Container (if so, version): N/A
Steps To Reproduce
-
Get pytorch weights of modified model
-
Transform them to onnx using deform-conv2d-onnx-exporter and export_model.py as it is said in detectron2. I want to clarify that batch is not dynamic, it is strictly 8. 2.1. As a result of running this scrpt on image of size (3x512x910) I get the following ONNX file 2.2. If I run the same script on batch of images of size (8x3x512x910) I get the other ONNX
-
After it I an trying to transform above mentioned onnx files to .engine files using the following command
/usr/src/tensorrt/bin/trtexec --onnx=./bs_XXX.onnx --workspace=2048 --explicitBatch --saveEngine=./bs_XXX.engine --fp16 -
After I am trying to inference them using this script It works for batch_size=1, but not for batch_size=8.
I tried converting .onnx file of model with batch_size=1 and after using /usr/src/tensorrt/bin/trtexec --onnx=./bs_XXX.onnx --workspace=2048 command with --batch flag or --maxBatch, but trt fails.
Have you tried the latest release?: Yes
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes
I guess your onnx model only support batch size it. what is the model input shape? what error do you see?
closing since no activity for more than 3 weeks, pls reopen if you still have question, thanks!
