🐛 [Bug] Shape analysis failure when encountered dynamic fallback

[Njuapp]

Bug Description

In dynamic fallback scenarios, the shape analysis will fail in generateRandomInputs phase.

Traceback (most recent call last):
  File "main.py", line 56, in <module>
    main()
  File "main.py", line 37, in main
    model = torch_tensorrt.compile(torch_script_module.half(), **compile_settings)
  File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 115, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 113, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: Trying to create tensor with negative dimension -1: [-1, 3, 224, 224]

To Reproduce

Run the following code.

import torch
import torch_tensorrt
import torchvision
import time

torch_tensorrt.logging.set_reportable_log_level(torch_tensorrt.logging.Level.Debug)

warmup_time = 10
test_time = 100
def main():
    torch.manual_seed(2022)
    im_rand = torch.rand((16, 3, 224, 224)).cuda()
    resnet_model = torchvision.models.resnet50()
    resnet_model = resnet_model.eval().cuda()
    torch_script_module = torch.jit.trace(resnet_model, (im_rand))
    # torch.jit.save(torch_script_module.half(), 'resnet18.pt')

    # torch_script_module = torch.jit.load('resnet.pt')
    input = torch.rand((32, 3, 224, 224)).cuda()
    ts_output = torch_script_module(input)
    print('result:', ts_output)

    compile_settings = {
        # "inputs": [input],
        "inputs": [torch_tensorrt.Input(
            min_shape=[1, 3, 224, 224],  # TODO: depends on the model size
            opt_shape=[32, 3, 224, 224],
            max_shape=[32, 3, 224, 224],
            # shape=[1, 3, 224, 224],
            dtype=torch.half,  # Datatype of input tensor. Allowed options torch.(float|half|int8|int32|bool)
        )],
        'torch_executed_ops': ['aten::adaptive_avg_pool2d'],
        "enabled_precisions": {torch.half}
    }

    print("Compile: FP16")
    model = torch_tensorrt.compile(torch_script_module.half(), **compile_settings)

    for i in range(warmup_time):
        trt_output = model(input.half())

    print("Compile: FP16")
    t1 = time.time()
    for i in range(test_time):
        trt_output = model(input.half())
    t2 = time.time()
    print('trt_output: ', trt_output)

    diff = abs(ts_output - trt_output)
    print('diff: {}'.format(diff.mean()))


if __name__ == '__main__':
    main()

Expected behavior

The compilation could run successfully.

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• Torch-TensorRT Version (e.g. 1.1.0):
• PyTorch Version (e.g. 1.11):
• CPU Architecture: x86_64
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, libtorch, source): pip
• Build command you used (if compiling from source): bazel
• Python version: 3.8
• CUDA version: 11.3

[edric1261234]

same issue

[Njuapp]

@edric1261234 You can try if this PR can help: https://github.com/pytorch/TensorRT/pull/1111

[edric1261234]

@edric1261234 You can try if this PR can help: #1111

Thank you, it works!

[mallorbc]

Pretty sure I also have this issue. Trying to use dynamic input sizes for GPT models. I will look at PR later to see if that helps.

[dathudeptrai]

@narendasan can you take a look?

[narendasan]

In Torch-TRT 1.3.0 there is now support for dynamic shape with partial compilation enabled. Please try it out and let us know if it works. Reopen if the issue persists.