onnxruntime_backend
onnxruntime_backend copied to clipboard
Possible to enable dynamic batch dimension only on one some input tensors?
Context: I created a PyTorch model (nn.module) and exported to *.onnx using torch.onnx.export() function with dynamic batch dimension enabled on only some of the input tensors. The output tensor shape should (logically) be [-1 , 3, 1024, 1024], namely it should have dynamic batch dimension.
However, when trying to load the model in Triton, I got the following 2 errors:
- max_batch_size > 0 in config.pbtxt Seems Triton expects all the input tensors to have dynamic batch dimension, even though I only specified some of them when exporting the model.
- max_batch_size = 0 in config.pbtxt In this case, I can specify the dims for input tensors explicitly as expected. But for output tensor, I have to specify the dims as [-1,-1,-1,-1] to successfully load it. Otherwise, if I set it to [1, 3, 1024, 1024], I got error message: the model expects 4 dimensions (shape [-1,-1,-1,-1]) but the model configuration specifies 4 dimensions (shape [1,3,1024,1024])
Question:
- Is this by design? Any way I can enable dynamic batch dimension on only some of the input tensors?
- Is this the expected behavior? Why does the color channel & image size dimensions (3, 1024, 1024) are expected to be -1?
Thanks!