Tried with max_batch_size or defining min and max shapes but only one batch is possible.

[YOONAHLEE]

dynamic batch is not possible even though I defined min and max shapes or defined max_batch_size.

    x = torch.rand([1,1,5,5,2000]).cuda().half()
    model = model.half()
    model = torch2trt(model, [x], fp16_mode=True, max_batch_size=10)
    y = model(x)
    print('trt model sucess with one batch ', x.shape, y.shape)

I defined max batch size but only one batch was successfully passed.

batch size is  10
trt model sucess with one batch  torch.Size([1, 1, 5, 5, 2000]) torch.Size([1, 2])

if I passed with 2~10 batch size inputs, then it made the below error.

    # this fails
    y = model(torch.rand([10, 1, 5, 5, 2000]).cuda().half())
    print('trt model success 10 batch', y.shape)

    y = model(torch.rand([10, 1, 5, 5, 2000]).cuda().half())
  File "/home/user_name/miniconda3/envs/deploy/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user_name/miniconda3/envs/deploy/lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 593, in forward
    shape = tuple(self.context.get_binding_shape(idx))
ValueError: __len__() should return >= 0

So I tried different parameters. I defined min_shapes and max_shapes. However, torch2trt function couldn't support these parameters.

    model = torch2trt(module=model, inputs=[x], min_shapes=[(1, 1, 5, 5, 2000)], max_shapes=[(10, 1, 5, 5, 2000)], opt_shapes=[(10, 1, 5, 5, 2000)], fp16_mode=True)
  File "/home/user_name/miniconda3/envs/deploy/lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 739, in torch2trt
    ctx.mark_outputs(outputs_flat, output_names)
  File "/home/user_name/miniconda3/envs/deploy/lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 521, in mark_outputs
    trt_tensor = torch_output._trt
AttributeError: 'Tensor' object has no attribute '_trt'

Are these parameters available?

[YOONAHLEE]

I read the previous issue (https://github.com/NVIDIA-AI-IOT/torch2trt/issues/740) and run the #779 pull request code. This made max_batch_size parameter work.

However, when I pass min_shapes and max_shapes parameters, I got the same above error.

Also, in "torch2trt/converters/getitem.py", undefined "make_int_wrapper" function was used. It makes undefined nameerror.

[jaybdub]

Hi @YOONAHLEE ,

Thanks for reaching out!

I think this was an overlook on my part, I've added function make_int_wrapper to the file to the PR.

Do you mind trying now?

Best, John

[YOONAHLEE]

Hi @jaybdub Thank you for answering the question!

I've got the another error when converting torch.nn.BatchNorm2d layer in the resnet18 model. (self.bn1 layer is torch.nn.BatchNorm2d layer here) Do you know what makes this error?

[TRT] [E] Could not implicitly convert NumPy data type: i64 to TensorRT.


   return self._forward_impl(x)
  File "~model_code.py", line 190, in _forward_impl
    x = self.bn1(x)
  File "/~lib/python3.9/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/~lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 148, in forward
    self.num_batches_tracked = self.num_batches_tracked + 1  # type: ignore[has-type]
  File "/~lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 307, in wrapper
    converter["converter"](ctx)
  File "/~lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/converters/add.py", line 13, in convert_add
    input_a_trt, input_b_trt = add_missing_trt_tensors(ctx.network, [input_a, input_b])
  File "/~lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 175, in add_missing_trt_tensors
    t._trt = network.add_constant(shape, weight).get_output(0)
  File "/~lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 400, in wrapper
    ret = attr(*args, **kwargs)
TypeError: add_constant(): incompatible function arguments. The following argument types are supported:
    1. (self: tensorrt.tensorrt.INetworkDefinition, shape: tensorrt.tensorrt.Dims, weights: tensorrt.tensorrt.Weights) -> tensorrt.tensorrt.IConstantLayer

I didn't call

model.eval()

before compiling the model. After I called model.eval(), the above error was solved. However, still I get the error when converting torch.nn.Conv2d layer

# print(module)
Conv2d(1, 32, kernel_size=(7, 20), stride=(2, 2), padding=(3, 0), bias=False)
[08/11/2022-05:14:50] [TRT] [E] 3: [network.cpp::addConvolutionNd::990] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/network.cpp::addConvolutionNd::990, condition: containsTensor(inputTensor))


  File "/~lib/python3.9/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/~lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/torch2trt.py", line 307, in wrapper
    converter["converter"](ctx)
  File "/~lib/python3.9/site-packages/torch2trt-0.4.0-py3.9.egg/torch2trt/converters/Conv.py", line 43, in convert_Conv_trt7
    layer.stride_nd = stride
AttributeError: 'NoneType' object has no attribute 'stride_nd'

[YOONAHLEE]

I solved the above problem by defining a new input. I could see that input was not passed well into the converter function.

I was using multiple models and I converted like below

x = torch.rand((shape)).cuda()
model1 = torch2trt(model1, [x], max_batch_size=100)
model2 = torch2trt(model2, [x], max_batch_size=100)

and This made the problem in the first nn.conv2d layer in model2.

after I defined the input tensor again, input tensor was passed well in convert function.

x = torch.rand((shape)).cuda()
mode1 = torch2trt(model1, [x], max_batch_size=100)
x = torch.rand((shape)).cuda()
model2 = torch2trt(model2, [x], max_batch_size=100)

I guess convert function changes the input

[A-janjan]

thanks a lot, bro.

but why?