TensorRT icon indicating copy to clipboard operation
TensorRT copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

✨[Feature] Bilinear Model Support

Open agemagician opened this issue 3 years ago • 4 comments

Is your feature request related to a problem? Please describe. Not related to a problem

Describe the solution you'd like I have a model that has a Bilinear layer and it can not be converted to TensorRT.

Describe alternatives you've considered Changing the Bilinear to anything else doesn't provide the same model accuracy.

Additional context

ERROR: [Torch-TensorRT] - Unsupported operator: aten::bilinear(Tensor input1, Tensor input2, Tensor weight, Tensor? bias) -> (Tensor)
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py(1974): bilinear
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py(189): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/tmp/ipykernel_1021/3297051232.py(67): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/tmp/ipykernel_1021/1097032733.py(3): <module>
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3457): run_code
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3377): run_ast_nodes
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3185): run_cell_async
/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2960): _run_cell
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2914): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py(533): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py(353): do_execute
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(648): execute_request
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(353): dispatch_shell
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(446): process_one
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(457): dispatch_queue
/opt/conda/lib/python3.8/asyncio/events.py(81): _run
/opt/conda/lib/python3.8/asyncio/base_events.py(1859): _run_once
/opt/conda/lib/python3.8/asyncio/base_events.py(570): run_forever
/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py(199): start
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py(677): start
/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py(846): launch_instance
/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py(16): <module>
/opt/conda/lib/python3.8/runpy.py(87): _run_code
/opt/conda/lib/python3.8/runpy.py(194): _run_module_as_main

ERROR: [Torch-TensorRT] - Unsupported operator: aten::gather(Tensor self, int dim, Tensor index, *, bool sparse_grad=False) -> (Tensor)
/tmp/ipykernel_1021/3297051232.py(55): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/tmp/ipykernel_1021/1097032733.py(3): <module>
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3457): run_code
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3377): run_ast_nodes
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3185): run_cell_async
/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2960): _run_cell
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2914): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py(533): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py(353): do_execute
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(648): execute_request
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(353): dispatch_shell
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(446): process_one
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(457): dispatch_queue
/opt/conda/lib/python3.8/asyncio/events.py(81): _run
/opt/conda/lib/python3.8/asyncio/base_events.py(1859): _run_once
/opt/conda/lib/python3.8/asyncio/base_events.py(570): run_forever
/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py(199): start
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py(677): start
/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py(846): launch_instance
/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py(16): <module>
/opt/conda/lib/python3.8/runpy.py(87): _run_code
/opt/conda/lib/python3.8/runpy.py(194): _run_module_as_main
/tmp/ipykernel_1021/3297051232.py(44): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/tmp/ipykernel_1021/1097032733.py(3): <module>
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3457): run_code
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3377): run_ast_nodes
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3185): run_cell_async
/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2960): _run_cell
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2914): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py(533): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py(353): do_execute
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(648): execute_request
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(353): dispatch_shell
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(446): process_one
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(457): dispatch_queue
/opt/conda/lib/python3.8/asyncio/events.py(81): _run
/opt/conda/lib/python3.8/asyncio/base_events.py(1859): _run_once
/opt/conda/lib/python3.8/asyncio/base_events.py(570): run_forever
/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py(199): start
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py(677): start
/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py(846): launch_instance
/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py(16): <module>
/opt/conda/lib/python3.8/runpy.py(87): _run_code
/opt/conda/lib/python3.8/runpy.py(194): _run_module_as_main

ERROR: [Torch-TensorRT] - Method requested cannot be compiled by Torch-TensorRT.TorchScript.
Unsupported operators listed below:
  - aten::bilinear(Tensor input1, Tensor input2, Tensor weight, Tensor? bias) -> (Tensor)
  - aten::gather(Tensor self, int dim, Tensor index, *, bool sparse_grad=False) -> (Tensor)
You can either implement converters for these ops in your application or request implementation
https://www.github.com/nvidia/Torch-TensorRT/issues

In Module:

ERROR: [Torch-TensorRT] - Unsupported operator: aten::bilinear(Tensor input1, Tensor input2, Tensor weight, Tensor? bias) -> (Tensor)
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py(1974): bilinear
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py(189): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/tmp/ipykernel_1021/3297051232.py(67): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(9

agemagician avatar Jan 25 '22 16:01 agemagician

@agemagician Just because there is a known bug with the traceback that might produce misleading errors, are you trying to use the partial compilation feature?

narendasan avatar Jan 25 '22 16:01 narendasan

Here is a small example that re-produce the problem of my big model:

m = torch.nn.Bilinear(20, 30, 40).to("cuda")

input1 = torch.randn(128, 20).to("cuda")
input2 = torch.randn(128, 30).to("cuda")

output = m(input1, input2).to("cuda")

traced_model = torch.jit.trace(
    m,
    [input1,
     input2
    ]
)

trt_model_fp32 = torch_tensorrt.compile(traced_model, **{
    "inputs":
        [
            torch_tensorrt.Input((input1.shape[0], input1.shape[1]), dtype=torch.float32),
            torch_tensorrt.Input((input1.shape[0], input1.shape[1]), dtype=torch.float32),
        ],
    "enabled_precisions": {torch.float32}, # Run with FP32
    "workspace_size": 1 << 22
})

Error Logs:

ERROR: [Torch-TensorRT] - Unsupported operator: aten::bilinear(Tensor input1, Tensor input2, Tensor weight, Tensor? bias) -> (Tensor)
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py(1974): bilinear
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py(189): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/tmp/ipykernel_1021/3155311369.py(1): <module>
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3457): run_code
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3377): run_ast_nodes
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3185): run_cell_async
/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2960): _run_cell
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2914): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py(533): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py(353): do_execute
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(648): execute_request
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(353): dispatch_shell
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(446): process_one
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(457): dispatch_queue
/opt/conda/lib/python3.8/asyncio/events.py(81): _run
/opt/conda/lib/python3.8/asyncio/base_events.py(1859): _run_once
/opt/conda/lib/python3.8/asyncio/base_events.py(570): run_forever
/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py(199): start
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py(677): start
/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py(846): launch_instance
/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py(16): <module>
/opt/conda/lib/python3.8/runpy.py(87): _run_code
/opt/conda/lib/python3.8/runpy.py(194): _run_module_as_main

ERROR: [Torch-TensorRT] - Method requested cannot be compiled by Torch-TensorRT.TorchScript.
Unsupported operators listed below:
  - aten::bilinear(Tensor input1, Tensor input2, Tensor weight, Tensor? bias) -> (Tensor)
You can either implement converters for these ops in your application or request implementation
https://www.github.com/nvidia/Torch-TensorRT/issues

In Module:

ERROR: [Torch-TensorRT] - Unsupported operator: aten::bilinear(Tensor input1, Tensor input2, Tensor weight, Tensor? bias) -> (Tensor)
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py(1974): bilinear
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py(189): forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/tmp/ipykernel_1021/3155311369.py(1): <module>
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3457): run_code
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3377): run_ast_nodes
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3185): run_cell_async
/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2960): _run_cell
/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2914): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py(533): run_cell
/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py(353): do_execute
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(648): execute_request
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(353): dispatch_shell
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(446): process_one
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py(457): dispatch_queue
/opt/conda/lib/python3.8/asyncio/events.py(81): _run
/opt/conda/lib/python3.8/asyncio/base_events.py(1859): _run_once
/opt/conda/lib/python3.8/asyncio/base_events.py(570): run_forever
/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py(199): start
/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py(677): start
/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py(846): launch_instance
/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py(16): <module>
/opt/conda/lib/python3.8/runpy.py(87): _run_code
/opt/conda/lib/python3.8/runpy.py(194): _run_module_as_main

WARNING: [Torch-TensorRT] - Input type for doing shape analysis could not be determined, defaulting to F32
WARNING: [Torch-TensorRT] - Input type for doing shape analysis could not be determined, defaulting to F32
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_1021/1005173159.py in <module>
      3 # The compiled module will have precision as specified by "op_precision".
      4 # Here, it will have FP16 precision.
----> 5 trt_model_fp32 = torch_tensorrt.compile(traced_model, **{
      6     "inputs":
      7         #[torch_tensorrt.Input((128, 3, 224, 224), dtype=torch.float32)],

/opt/conda/lib/python3.8/site-packages/torch_tensorrt/_compile.py in compile(module, ir, inputs, enabled_precisions, **kwargs)
     95             )
     96             ts_mod = torch.jit.script(module)
---> 97         return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
     98     elif target_ir == _IRType.fx:
     99         raise RuntimeError("fx is currently not supported")

/opt/conda/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py in compile(module, inputs, device, disable_tf32, sparse_weights, enabled_precisions, refit, debug, strict_types, capability, num_min_timing_iters, num_avg_timing_iters, workspace_size, max_batch_size, calibrator, truncate_long_and_double, require_full_compilation, min_block_size, torch_executed_ops, torch_executed_modules)
    117     }
    118 
--> 119     compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
    120     compiled_module = torch.jit._recursive.wrap_cpp_module(compiled_cpp_mod)
    121     return compiled_module

RuntimeError: [Error thrown at core/partitioning/shape_analysis.cpp:67] Expected ivalues_maps.count(input) to be true but got false
Could not find torch::jit::Value* 6 produced from %input1 : Tensor, %input2 : Tensor, %6 : Float(40, 20, 30, strides=[600, 30, 1], requires_grad=1, device=cuda:0), %7 : Float(40, strides=[1], requires_grad=1, device=cuda:0) = prim::Param() in lowering graph for mini graph input.

agemagician avatar Jan 25 '22 17:01 agemagician

Jit Trace works fine, the error comes from tensorrt compile.

Any ideas how to make it work ?

agemagician avatar Jan 25 '22 17:01 agemagician

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

github-actions[bot] avatar Aug 17 '22 00:08 github-actions[bot]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

github-actions[bot] avatar Nov 21 '22 00:11 github-actions[bot]

@agemagician there was a fix for this class of issue in v1.3.0. Please try it out and let us know if it works for you, reopen if the issue persists.

narendasan avatar Dec 15 '22 16:12 narendasan