🐛 [Bug] Cannot convert any detection model from torchvision nor detectron

[batrlatom]

Bug Description

Hi, I have hard times to export retinanet or mask-rcnn. I successfully traced models both from torchvision and detectron, but it fails to compile. Since there are others who has problems with torchvision models, I would be happy if you could resolve this problem once and for all. The only model for object detection which works is SSD from nvidia torch hub, but supporting standart torchvision models would be a huge win.

To Reproduce

Steps to reproduce the behavior: Note that traced_model works well, no problem here

import torch
import torchvision
import torch_tensorrt

model = torchvision.models.detection.retinanet_resnet50_fpn(num_classes=1, pretrained=False)
model = model.eval().to("cuda")

traced_model = torch.jit.trace(model, torch.randn((1, 3, 640, 640)).to("cuda"), strict=False)
torch.jit.save(traced_model, "resnet_50_traced.jit.pt")

trt_model_fp32 = torch_tensorrt.compile(traced_model, **{
    "inputs": [torch.ones((1, 3, 640, 640), dtype=torch.float32).cuda()],
    "enabled_precisions": {torch.float32}, # Run with FP32
    "workspace_size": 1 << 22
})

Error: """ Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions """

Expected behavior

I expect it to work, like magic :) !

Environment

Standart docker build from your dockerfile, base pytorch image is 21.10

Additional context

Do you think you could provide us with the tutorial how to deploy at least one detection model from torchvision or detectron? end to end? Thank you!

[narendasan]

This seems like an error coming from torch.jit.trace, can you try using torch.jit.script?

[batrlatom]

If I try this code for scripting:

import torch
import torchvision
import torch_tensorrt

model = torchvision.models.detection.retinanet_resnet50_fpn(num_classes=1, pretrained=False)
model = model.eval().to("cuda")

scripted_model = torch.jit.script(model, torch.randn((1, 3, 640, 640)).cuda())
torch.jit.save(scripted_model, "resnet_50_scripted.jit.pt")

out = scripted_model([torch.randn((3, 640, 640), dtype=torch.float32).cuda()])
print(out)

trt_model_fp32 = torch_tensorrt.compile(scripted_model, **{
    "inputs": [torch.ones((3, 640, 640), dtype=torch.float32)],
    "enabled_precisions": {torch.float32}, # Run with FP32
    "workspace_size": 1 << 22
})

I am getting an error comming from compile: """ temporary: the only valid use of a module is looking up an attribute but found = prim::SetAttr[name="_has_warned"](%self, %self.backbone.fpn.extra_blocks.use_P5) """

Scripted model itself looks to work well, print(out) gets me """ ({}, [{'boxes': tensor([], device='cuda:0', size=(0, 4), grad_fn=<StackBackward0>), 'scores': tensor([], device='cuda:0', grad_fn=<IndexBackward0>), 'labels': tensor([], device='cuda:0', dtype=torch.int64)}]) """

Do you think your team could look into the support/tutorial for torchvision or detectron models? I am literally trying to compile retinanet and maskrcnn for over a year now. This would help a lot of people I think

[narendasan]

Yes, we are working on an end to end demo for detection models

[batrlatom]

I look forward to. thanks a lot!

[batrlatom]

Hi @narendasan , just asking... do you have any timetable for ''' end to end demo for detection models ''' ? is it possible that you can get into that this year?

[narendasan]

There isnt a hard timeline but we are actively working on it

[SnowRipple]

@narendasan any news on this?

[ncomly-nvidia]

We are in the process of supporting Detectron2 OD networks. We will updated when there is a concrete timeline.

[joshuafc]

Same problem, I have try FasterRCNN in torchvision.models.detection

install torch-tensorrt by pip3 install torch-tensorrt -f https://github.com/NVIDIA/Torch-TensorRT/releases

if use torch.jit.trace, when compile, """ Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions """

if use torch.jit.script, when compile, """ temporary: the only valid use of a module is looking up an attribute but found = prim::SetAttr[name="****"] """

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[joshuafc]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

Any news?

[peri044]

@batrlatom @joshuafc We had our focus on transformers and dynamic shape models in the recent times. We couldn't get to finish an example based on detectron @bowang007 Can you share the status of retinanet or mask-rcnn if you have worked on either of those ?

[Christina-Young-NVIDIA]

Model / feature request.

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[joshuafc]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

Any news?

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[joshuafc]

So we are all playing with LLMs and ChatGPT now?

[narendasan]

There are technical limitations upstream (torchscript, torchdynamo) that we are working with PyTorch on being resolved so nothing about detectron support has changed that much but it is still on our radar.