🐛 [Bug] Dynamic shape MobileNet compilation files

[pkluska]

Bug Description

When trying to compile MobileNetV3 from torchvision with the dynamic shape, it fails with following error

DEBUG: [Torch-TensorRT] - Pairing 0: x.1 : Input(shape: [-1, 3, 224, 224], min: [1, 3, 224, 224], opt: [64, 3, 224, 224], max: [128, 3, 224, 224], dtype: Float32, format: NCHW\Contiguous\Linear)
INFO: [Torch-TensorRT] - Since input type is not explicitly defined, infering using first tensor calculation
  Found input x.1 has type Float. If this is incorrect explicitly set dtype for input and file a bug
INFO: [Torch-TensorRT] - Method requested cannot be compiled end to end by Torch-TensorRT.TorchScript.
Unsupported operators listed below:
  - aten::hardsigmoid(Tensor self) -> (Tensor)

DEBUG: [Torch-TensorRT] - Unsupported operator: aten::hardsigmoid(Tensor self) -> (Tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1932
    if inplace:
        return torch._C._nn.hardsigmoid_(input)
    return torch._C._nn.hardsigmoid(input)
           ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

Traceback (most recent call last):
  File "example.py", line 23, in <module>
    main()
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "example.py", line 12, in main
    trt_model = trt.compile(model,
  File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 115, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 113, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: Trying to create tensor with negative dimension -1: [-1, 3, 224, 224

To Reproduce

Steps to reproduce the behavior:

1. Fetch MobileNetV3_Small model from torchvision
2. Try to compile with dynamic batch size.

import torch
from torchvision.models import mobilenet
import torch_tensorrt as trt


@torch.no_grad()
def main():
    model = mobilenet.mobilenet_v3_small(
        weights=mobilenet.MobileNet_V3_Small_Weights.DEFAULT)
    model = model.eval().to("cuda:0")
    with trt.logging.debug():
        trt_model = trt.compile(model,
                                inputs=[
                                    trt.Input(min_shape=(1, 3, 224, 224),
                                              opt_shape=(64, 3, 224, 224),
                                              max_shape=(128, 3, 224, 224))
                                ],
                                enabled_precisions={torch.float})
    print(trt_model)


if __name__ == "__main__":
    main()

Expected behavior

MobileNetV3 is compiled and can accept various batch sizes. The issue does not happen when using static shape:

import torch
from torchvision.models import mobilenet
import torch_tensorrt as trt


@torch.no_grad()
def main():
    model = mobilenet.mobilenet_v3_small(
        weights=mobilenet.MobileNet_V3_Small_Weights.DEFAULT)
    model = model.eval().to("cuda:0")
    with trt.logging.debug():
        trt_model = trt.compile(model,
                                inputs=[trt.Input(shape=(1, 3, 224, 224))],
                                enabled_precisions={torch.float})
    print(trt_model)


if __name__ == "__main__":
    main()

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• Docker Image: nvcr.io/nvidia/pytorch:22.06-py3
• Torch-TensorRT Version (e.g. 1.0.0): 1.1.0a
• PyTorch Version (e.g. 1.0): 1.13.0a0
• CPU Architecture: x86
• OS (e.g., Linux): Ubuntu
• How you installed PyTorch (conda, pip, libtorch, source): Docker from NGC
• Build command you used (if compiling from source): N/A
• Are you using local sources or building from archives: No
• Python version: 3.8
• CUDA version: 11.7
• GPU models and configuration: V100
• Any other relevant information:

[narendasan]

We will add hardsigmoid to the backlog for converters.

[peri044]

We currently don't support dynamic shape + fallback. During fallback, we generate random inputs of static shape to perform shape analysis and it fails here since the shape has -1 in its dimensions. Fallback path gets executed by default because there is an unsupported operator hard_sigmoid

[pkluska]

Thank you very much for the response!

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[ncomly-nvidia]

@peri044 for DB + fallback.

[narendasan]

@pkluska Can you try v1.3.0 which has this Dynamic batch feature / partial compilation feature? Reopen if it is still an issue.