❓ [Question] How can I add torchvision.transforms.functional.gaussian_blur to the conversion?

[mjack3]

❓ Question

How could I make torchvision.transforms.functional.gaussian_blur operation compatible with torch_tensorrt ?

What you have already tried

Hello. The last step of my forward method is to apply gaussian_blur. Unfortunately this is not compatible with this framework and I must to put it out of the forward method. If I do, the model is correctly parsed to TensorRT engine. If not, I get this error

ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: (Unnamed Layer* 613) [Convolution]: two inputs (data and weights) are allowed only in explicit-quantization mode.
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [network.cpp::validateWeightedLayersInputs::2378] Error Code 4: Internal Error ((Unnamed Layer* 613) [Convolution]: Cannot set more than one input unless network has Q/DQ layers.)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
Traceback (most recent call last):
  File "/home/jack3/tkh-projects/02-AD/code/TKHAD/kk.py", line 78, in <module>
    trt_model = torch_tensorrt.compile(
  File "/home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torch_tensorrt/_compile.py", line 115, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torch_tensorrt/ts/_compiler.py", line 113, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/conversion/conversionctx/ConversionCtx.cpp:147] Building serialized network failed in TensorRT

I suppose there is any operation inside gaussian_blur incompatible, although the error is not clear for me.

This is the code that convert the model

dummy_input = torch.empty((1, 3, 224 ,224), device=torch.device('cuda'))
jit_model = torch.jit.trace(model, dummy_input)

trt_model = torch_tensorrt.compile(
    jit_model,
    "default",
    [torch_tensorrt.Input((1, 3, 224, 224), dtype=torch.float32)],
    torch.float32,
    truncate_long_and_double = True
)

And this is the part of my model with gaussian_blur

from torchvision.transforms.functional import gaussian_blur
import torch

class MyModel(torch.nn.Module):
   def __init__(self):
   ... 
   self.kernel = 2 * int(4.0 * 4 + 0.5) + 1

    def forward(self, x: torch.Tensor):
        ...
        
        map_scores = gaussian_blur(map_scores, [self.kernel , self.kernel ], [4, 4])
        return map_scores

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• PyTorch Version (e.g., 1.0): 1.11.0+cu113
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, libtorch, source): pip
• Python version: 3.10
• CUDA version: 11.7
• Torch_tensorrt Version: 1.1.0

[narendasan]

Hmm, I'll have to look into the implementation, but theres probably some conv that takes in to tensors which are both produced during the course of computation. It would help if you could turn on debug logging and share what the graph looks like and the output of the conv converter when it runs on the ops for Gaussian blur. Also, you could instruct torch_tensorrt to run all instances of the gaussian blur class to run in pytorch using torch_executed_modules

[mjack3]

Ok, I go for it

[mjack3]

Hello @narendasan I will organize the answers:

+ It would help if you could turn on debug logging

This is how ends the code of my foward method

all_likelihoods = torch.cat(all_likelihoods, 0) # concat results
map_scores = all_likelihoods.mean(0, keepdim=True) # mean of my results
map_scores = self.gaussian_blur(map_scores) # here is the problem
return map_scores # end

Here the output of using torch_tensorrt.logging.debug. I will share the part where gaussian_blur appears, if not the text would be very long.

 %all_likelihoods : Tensor = aten::cat(%392, %329) # /home/jack3/tkh-projects/02-AD/code/TKHAD/modules/model.py:82:0
  %img.1 : Tensor = aten::mean(%all_likelihoods, %330, %15, %20) # /home/jack3/tkh-projects/02-AD/code/TKHAD/modules/model.py:82:0
  %395 : int = aten::size(%img.1, %18), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:766:0
  %396 : int[] = prim::ListConstruct(%395, %18, %331, %331), scope: __module.gaussian_blur
  %kernel : Tensor = aten::expand(%332, %396, %17), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:766:0
  %img : Tensor = aten::reflection_pad2d(%img.1, %333), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torch/nn/functional.py:4367:0
  %161 : Tensor = aten::_convolution(%img, %kernel, %20, %12, %11, %12, %17, %11, %18, %17, %17, %15, %15), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:768:0
  return (%161)
..
..
WARNING: [Torch-TensorRT] - Mean converter disregards dtype
DEBUG: [Torch-TensorRT] - Output shape: [1, 1, 224, 224]
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Evaluating %395 : int = aten::size(%img.1, %18), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:766:0
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Found the value to be: 1
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Evaluating %396 : int[] = prim::ListConstruct(%395, %18, %331, %331), scope: __module.gaussian_blur
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Found the value to be: [1, 1, 33, 33]
INFO: [Torch-TensorRT TorchScript Conversion Context] - Adding Layer %kernel : Tensor = aten::expand(%332, %396, %17), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:766:0 (ctx.AddLayer)
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Found IValue containing object of type Float(33, 33, strides=[33, 1], requires_grad=0, device=cuda:0)
..
..
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Freezing tensor 0x5618c0379cc0 as an IConstantLayer
DEBUG: [Torch-TensorRT] - ITensor shape: [33, 33]
DEBUG: [Torch-TensorRT] - ITensor type: Float32
DEBUG: [Torch-TensorRT] - (expand layer) Expand input from [33, 33] to [1, 1, 33, 33]
DEBUG: [Torch-TensorRT] - Input reshaped to : [1, 1, 33, 33] from [33, 33]
DEBUG: [Torch-TensorRT] - Expand layer output tensor shape: [1, 1, 33, 33]
INFO: [Torch-TensorRT TorchScript Conversion Context] - Adding Layer %img : Tensor = aten::reflection_pad2d(%img.1, %333), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torch/nn/functional.py:4367:0 (ctx.AddLayer)
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is an already converted tensor
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT] - Weights: [1]
    Data Type: Int32
    Number of input maps: 1
    Number of output maps: 1
    Element shape: [1]
...
...
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Freezing tensor 0x5618c04f2170 as an IConstantLayer
DEBUG: [Torch-TensorRT] - Output tensor shape: [1, 1, 256, 256]
INFO: [Torch-TensorRT TorchScript Conversion Context] - Adding Layer %161 : Tensor = aten::_convolution(%img, %kernel, %20, %12, %11, %12, %17, %11, %18, %17, %17, %15, %15), scope: __module.gaussian_blur # /home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:768:0 (ctx.AddLayer)
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is an already converted tensor
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is an already converted tensor
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Node input is a result of a previously evaluated value
DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Kernel weights are not set yet. Kernel weights must be set using setInput(1, kernel_tensor) API call.
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: (Unnamed Layer* 206) [Convolution]: two inputs (data and weights) are allowed only in explicit-quantization mode.
DEBUG: [Torch-TensorRT] - Output tensor shape: []
INFO: [Torch-TensorRT TorchScript Conversion Context] - Marking Output 161 named output_0 in engine (ctx.MarkOutput)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [network.cpp::validateWeightedLayersInputs::2378] Error Code 4: Internal Error ((Unnamed Layer* 206) [Convolution]: Cannot set more than one input unless network has Q/DQ layers.)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )

+ and the output of the conv converter when it runs on the ops for Gaussian blur

Input before gauss = [1, 1, 224, 224] Output after gauss = [1, 1, 224, 224]

I am not sure if this is what you mean.

+ you could instruct torch_tensorrt to run all instances of the gaussian blur class to run in pytorch using torch_executed_modules

I am trying it but I cannot figure out how can use it. 'torch_executed_modules (List[str]) – List of modules`

This is what I tried

for name, module in pytorch_model.named_modules():
    print("name:", name, "MODULE:", module)

Output:

name: decoder.0.module_list.3.subnet2.1 MODULE: ReLU()
name: decoder.0.module_list.3.subnet2.2 MODULE: Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1))
name: decoder.0.module_list.3.softplus MODULE: Softplus(beta=0.5, threshold=20)
name: gaussian_blur MODULE: GaussianBlur(kernel_size=(33, 33), sigma=(4.0, 4.0))

Then I tried both ["gaussian_blur"] and ["GaussianBlur(kernel_size=(33, 33), sigma=(4.0, 4.0))"] as a List[str] but I am getting an error:

Input

trt_model = torch_tensorrt.compile(
    traced_model,
    "default",
    [torch_tensorrt.Input((1, 3, 224, 224), dtype=torch.float32)],
    torch.float32,
    truncate_long_and_double = True,
    torch_executed_modules = ["gaussian_blur"], # list[str]
    require_full_compilation = False
)

Output:

Traceback (most recent call last):
  File "/home/jack3/tkh-projects/02-AD/code/TKHAD/kk.py", line 38, in <module>
    trt_model = torch_tensorrt.compile(
  File "/home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torch_tensorrt/_compile.py", line 115, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/home/jack3/tkh-projects/02-AD/code/TKHAD/env/lib/python3.10/site-packages/torch_tensorrt/ts/_compiler.py", line 113, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: Method 'forward1' is not defined.

---

I hope this can help you

[mjack3]

Is there any update about this?

• torchvision.transforms.gaussian_blur inside the forward method raises error?
• How can I use torch_executed_modules? Documentation about it is very poor

Thanks

[mjack3]

Hello. Could you give me a example of how to apply torch_executed_modules?

Thanks

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[mjack3]

Meh

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days