grimoire
Unable to generate mask results.
Hi, thanks to this project, I have managed to convert my mask_rcnn_r50_fpn_1x_coco trained using mmdet to tensorrt.
I have made a small test program that loads the pytorch model + config and converts it to trt before running inference. It is largely based on the demo program in demo/inference.py. See below:
from argparse import ArgumentParser
import cv2
import torch
from mmdet2trt import mmdet2trt
from mmdet2trt.apis import create_wrap_detector
from mmdet.apis import init_detector, inference_detector
import time
def main():
image = "testdata/testimg1/imageA009.png"
cfg_path = "mmdetection/configs/cfg/mask_rcnn_r50_fpn_1x_coco_logend_batch12_768_768_scale_n1000.py"
checkpoint = "mmdetection/work_dirs/mask_rcnn_r50_fpn_1x_coco_logend_batch12_768_768_scale_n1000/latest.pth"
fp16_mode = False
device = 0
save_path = "output.trt"
score_thr = 0.5
opt_shape_param = [
[
[1, 3, 160, 160], # min shape
[1, 3, 320, 320], # optimize shape
[1, 3, 768, 768], # max shape
]
]
#max_workspace_size = 1 << 30 # some module and tactic need large workspace.
if 0:
# generate trt model
trt_model = mmdet2trt(
cfg_path,
checkpoint,
fp16_mode=True,
int8_mode=False,
opt_shape_param=opt_shape_param,
trt_log_level='INFO',
output_names=['num_detections', 'boxes', 'scores', 'classes', 'masks'],
enable_mask=True)
# save converted model
torch.save(trt_model.state_dict(), save_path)
if 1:
#tensorrt
detector = create_wrap_detector(save_path, cfg_path, device)
else:
#pytorch
dev = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
detector = init_detector(cfg_path, checkpoint, device=dev)
_start_time = time.time()
result = inference_detector(detector, image)
duration = time.time() - _start_time
print("Prediction time is {}".format(duration))
detector.show_result(
image,
result,
score_thr=score_thr,
win_name='mmdet2trt_demo',
show=True)
if __name__ == '__main__':
main()
This appears to work well until I enable mask output. If I enable mask output using the "enable_mask=True", I get a number of errors:
Traceback (most recent call last):
File "/home/jo/projects/test/run_tensorrt_demo.py", line 84, in <module>
main()
File "/home/jo/projects/test/run_tensorrt_demo.py", line 71, in main
result = inference_detector(detector, image)
File "/home/jo/projects/test/mmdetection/mmdet/apis/inference.py", line 148, in inference_detector
results = model(return_loss=False, rescale=True, **data)
File "/home/jo/projects/test/venv-mmdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jo/projects/test/mmdetection-to-tensorrt/mmdet2trt/apis/inference.py", line 181, in forward
masks = masks.astype(np.float32)
AttributeError: 'Tensor' object has no attribute 'astype'
It appears that the error occurs due to some assumptions made about the type of "masks" - masks is a tensor - not a numpy array so we cannot use astype(np.float32). Maybe this works on a previous version of mmdet or maybe it has not been tested?
Anyways, if I fix the issue by converting to a float32 by calling masks = masks.type(torch.float32).cpu() (I had to use conventional RAM due to my limited GPU memory) the code continues but fails later:
Traceback (most recent call last):
File "/home/jo/projects/test/run_tensorrt_demo.py", line 84, in <module>
main()
File "/home/jo/projects/test/run_tensorrt_demo.py", line 71, in main
result = inference_detector(detector, image)
File "/home/jo/projects/test/mmdetection/mmdet/apis/inference.py", line 148, in inference_detector
results = model(return_loss=False, rescale=True, **data)
File "/home/jo/projects/test/venv-mmdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jo/projects/test/mmdetection-to-tensorrt/mmdet2trt/apis/inference.py", line 192, in forward
segms_results[labels[j]].append(masks[j])
TypeError: only integer tensors of a single element can be converted to an index
This issue is due to the type of "labels" as it contains floating point numbers which cannot be used for indexing.
labels = labels.type(torch.int) for j in range(len(dets)): segms_results[labels[j]].append(masks[j])
Converting it to an int beforehand solves the problem partially and a result plot is returned. However, it contains the existing detections with huge masks overlayed on top. So something isn't working properly when enabling masks. It looks like the masks themselves are 28x28 pixels large - even though the images are much larger.
Also, I am currently using fp16_mode due to limited CUDA memory available on my current PC. Maybe this is the issue?
How much of the mask functionality is working - and do you have any good ideas on how to proceed? I understand mask support is experimental so maybe I am overlooking a parameter that must be set.
MMDetection has changed a lot after I finish the mask export. I will try to fix it this weekend.
Mask output is 28*28 since I do not include the post-process of mask in the model. Grid-sample require too many memories. You can do post process by yourself like what MMDetection do.
Thanks for your suggestion. I manged to generate upscaled masks using mmdet get_seg_masks and did have some success. I replaced your rescale with the following - and now I get correct looking masks. One interesting note: If I set mask_thr_binary to 0.5 as set in my config, I get masks that contain a larger region around the object. If I bump it up to 0.6 - then it works as expected and the mask contains an output comparable to the pytorch model output.
Mask output works as expected with FP16 enabled/disabled but I have yet to get it working when using INT8. When enabling INT8, the returned masks appear very large/incorrect. I have never used INT8 before so maybe its not just a matter of enabling a flag...
if batch_masks is not None:
masks = batch_masks[i][:num_dets]
boxes = batch_boxes[i][:num_dets]
img_h, img_w = img_metas[i]['img_shape'][:2]
ori_h, ori_w = img_metas[i]['ori_shape'][:2]
masks = masks[:, :img_h, :img_w]
# upscale masks using mmdet
labels = labels.to(dtype=torch.int)
FCNMaskHeadInst = FCNMaskHead(num_classes=len(self.CLASSES), num_convs=0)
#rcnn_test_cfg = self.cfg._cfg_dict.model.test_cfg.rcnn
# Note: We have to use a higher threshold, otherwise masks contain garbage. Why?
rcnn_test_cfg = mmcv.Config({'mask_thr_binary': 0.6, })
masks = FCNMaskHeadInst.get_seg_masks(masks.unsqueeze(-3), boxes,
labels.cpu().numpy(),
rcnn_test_cfg,
img_metas[i]['ori_shape'][:2],
img_metas[i]['scale_factor'],
True)
results.append((dets_results, masks))
I see you already added your changes, thanks for that.
I noticed one minor bug. Disabling "rescale" results in the following:
Traceback (most recent call last):
File "run_tensorrt_demo.py", line 104, in <module>
main()
File "run_tensorrt_demo.py", line 72, in main
result = inference_detector(detector, image)
File "/home/nvidia/git/sscale/logends2/mmdetection/mmdet/apis/inference.py", line 147, in inference_detector
results = model(return_loss=False, rescale=False, **data)
File "/home/nvidia/git/sscale/logends2/venv-mmdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/nvidia/git/sscale/logends2/mmdetection-to-tensorrt/mmdet2trt/apis/inference.py", line 195, in forward
scale_factor=scale_factor,
UnboundLocalError: local variable 'scale_factor' referenced before assignment
Moving https://github.com/grimoire/mmdetection-to-tensorrt/blob/f1849cb9a2888554aba6d2d7e54c0876702233e3/mmdet2trt/apis/inference.py#L159 further up solves this problem
But unfortunately, the get_seg_masks approach is terribly slow (especially on Jetson, where inference takes significantly longer) so I will probably end up working on the raw 28x28 masks directly.
I have the same problem as you @tehkillerbee @grimoire . Using the "enable_mask=False" models can be generated.Using the "enable_mask=True" there are errors:
module mmdet.models.dense_heads.YOLOXHead not exist.
module mmdet.models.necks.YOLOXPAFPN not exist.
module mmdet.models.YOLOX not exist.
/home2/jyq/anaconda3/envs/mmdetrt/lib/python3.7/site-packages/mmdet/core/anchor/builder.py:16: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator``
'``build_anchor_generator`` would be deprecated soon, please use '
Use load_from_local loader
/home2/jyq/anaconda3/envs/mmdetrt/lib/python3.7/site-packages/mmdet/core/anchor/anchor_generator.py:360: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors``
'``single_level_grid_anchors`` would be deprecated soon. '
Warning: Encountered known unsupported method torch.Tensor.new_tensor
Warning: Encountered known unsupported method torch.Tensor.new_tensor
[TensorRT] INFO: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[TensorRT] INTERNAL ERROR: Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS
../rtSafe/cublas/cublasLtWrapper.cpp:279
Aborting...
[TensorRT] ERROR: ../rtSafe/cublas/cublasLtWrapper.cpp (279) - Assertion Error in getCublasLtHeuristic: 0 (cublasStatus == CUBLAS_STATUS_SUCCESS)
Traceback (most recent call last):
File "demo/inference_mask_rcnn.py", line 97, in <module>
main()
File "demo/inference_mask_rcnn.py", line 54, in main
torch.save(trt_model.state_dict(), checkpoin_save)
File "/home2/jyq/anaconda3/envs/mmdetrt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 909, in state_dict
hook_result = hook(self, destination, prefix, local_metadata)
File "/home2/jyq/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 452, in _on_state_dict
state_dict[prefix + 'engine'] = bytearray(self.engine.serialize())
AttributeError: 'NoneType' object has no attribute 'serialize'
Can you solve it for me? thank you!
I have the same problem as you @tehkillerbee @grimoire . Using the "enable_mask=False" models can be generated.Using the "enable_mask=True" there are errors:
module mmdet.models.dense_heads.YOLOXHead not exist. module mmdet.models.necks.YOLOXPAFPN not exist. module mmdet.models.YOLOX not exist. /home2/jyq/anaconda3/envs/mmdetrt/lib/python3.7/site-packages/mmdet/core/anchor/builder.py:16: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator`` '``build_anchor_generator`` would be deprecated soon, please use ' Use load_from_local loader /home2/jyq/anaconda3/envs/mmdetrt/lib/python3.7/site-packages/mmdet/core/anchor/anchor_generator.py:360: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors`` '``single_level_grid_anchors`` would be deprecated soon. ' Warning: Encountered known unsupported method torch.Tensor.new_tensor Warning: Encountered known unsupported method torch.Tensor.new_tensor [TensorRT] INFO: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [TensorRT] INTERNAL ERROR: Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS ../rtSafe/cublas/cublasLtWrapper.cpp:279 Aborting... [TensorRT] ERROR: ../rtSafe/cublas/cublasLtWrapper.cpp (279) - Assertion Error in getCublasLtHeuristic: 0 (cublasStatus == CUBLAS_STATUS_SUCCESS) Traceback (most recent call last): File "demo/inference_mask_rcnn.py", line 97, in <module> main() File "demo/inference_mask_rcnn.py", line 54, in main torch.save(trt_model.state_dict(), checkpoin_save) File "/home2/jyq/anaconda3/envs/mmdetrt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 909, in state_dict hook_result = hook(self, destination, prefix, local_metadata) File "/home2/jyq/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 452, in _on_state_dict state_dict[prefix + 'engine'] = bytearray(self.engine.serialize()) AttributeError: 'NoneType' object has no attribute 'serialize'Can you solve it for me? thank you!
I tried to update cuda10.2 Patch 1 and Patch 2, but it has not been solved.
@JingYQ I'm not sure this is the same issue I had. What versions of TensorRT are you using?
Do you have a piece of example code or did you use my example code directly?
I have made a small test program that loads the pytorch model + config and converts it to trt before running inference
@tehkillerbee Dear tehkillerbee if you set enable_mask=False Can it produce correct results?
@azuryl Yep, my example code works if I set enable_mask=False and output_names=['num_detections', 'boxes', 'scores', 'classes'],
It also works with masks enabled. Please note I have not tested it recently.