RTDETR

[MarcoPrassel]

Hi @marcoslucianops , I am trying to integrate Ultralytics' custom RT-DETR model with Deepstream, but I am having problems with the layer parser from ONNX. Could you please help me ? This is my ONNX's output

[rpatidar]

Did it work for you ?

[MarcoPrassel]

@rpatidar No, it did not work. I also tried to take advantage of the Nvidia Deepstream solutions, but the output and input layers have to be changed. I hope @marcoslucianops can help me.

[rpatidar]

Ohk , I am able to evaluate the standalone model with tensort - https://github.com/lyuwenyu/RT-DETR/blob/main/benchmark/trtinfer.py example shared here

thought tensorrt working should have allowed deepstream to work as well, looks like not working may need some customization.

looking for your expertise @marcoslucianops on how can we use the rt-detr model with the deepstream

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Initialized
0:00:00.613182162 48840 0x7feee4002460 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1174> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5

0:00:05.802209521 48840 0x7feee4002460 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/home/pranay/Desktop/model-rt-detr/rtdetr_r101vd_6x_coco_from_paddle.engine
0:00:05.822380989 48840 0x7feee4002460 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /home/pranay/Desktop/model-rt-detr/rtdetr_r101vd_6x_coco_from_paddle.engine
0:00:05.823421857 48840 0x7feee4002460 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initNonImageInputLayers() <nvdsinfer_context_impl.cpp:1542> [UID = 1]: More than one input layers but custom initialization function not implemented
0:00:05.823429100 48840 0x7feee4002460 ERROR                nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1316> [UID = 1]: Failed to initialize non-image input layers
0:00:05.841597013 48840 0x7feee4002460 WARN                 nvinfer gstnvinfer.cpp:898:gst_nvinfer_start:<primary-inference> error: Failed to create NvDsInferContext instance
0:00:05.841751157 48840 0x7feee4002460 WARN                 nvinfer gstnvinfer.cpp:898:gst_nvinfer_start:<primary-inference> error: Config file path: model_config/rtdetr_mode.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 5
0   INPUT  kFLOAT images          3x640x640       
1   INPUT  kINT32 orig_target_sizes 2               
2   OUTPUT kFLOAT scores          300             
3   OUTPUT kINT32 labels          300             
4   OUTPUT kFLOAT boxes           300x4           

[NvMultiObjectTracker] De-initialized

[marcoslucianops]

I will add the support today. For now, only for this repo: https://github.com/lyuwenyu/RT-DETR

Edit: I'm having issues with the bounding boxes output with the ultralytics model. It's not producing the bbox coordinates when the ONNX model is converted to TensorRT, but it produces the scores.

[marcoslucianops]

https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/RTDETR.md

[rpatidar]

I can confirms, I am able to use it, haven't tested all batching feature and other stuffs but for single stream its working as expected.

Thanks you so much. I was almost lost on how to get it working.

[tms2003]

@marcoslucianops I noticed that there is no benchmark yet. Does it perform well?

[rpatidar]

I was able to execute the ultralytics based model by adding following patch to export_yolov8.py

-    def __init__(self):
+    def __init__(self, img_size):
+        self.img_size = img_size
         super().__init__()

     def forward(self, x):
-        x = x.transpose(1, 2)
+        #x = x.transpose(1, 2)
         boxes = x[:, :, :4]
+        boxes[:, :, [0, 2]] *= self.img_size[1]
+        boxes[:, :, [1, 3]] *= self.img_size[0]
         scores, classes = torch.max(x[:, :, 4:], 2, keepdim=True)
         classes = classes.float()
         return boxes, scores, classes
@@ -64,7 +67,7 @@ def main(args):
             f.write(name + '\n')
         f.close()

-    model = nn.Sequential(model, DeepStreamOutput())
+    model = nn.Sequential(model, DeepStreamOutput((640,640)))

[marcoslucianops]

Thank you @rpatidar, the multiply to the img_size did the trick.

[citywatchnz]

Hi,

Has the RT-DETR been tested on the Jetson NX? When I try the a trained onnx model on the Jetson NX, the conversion failed with the below message. It works fine on the PC - converted and running. I am using Deepstream 6.0.

WARNING: [TRT]: onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. WARNING: [TRT]: onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped ERROR: [TRT]: ModelImporter.cpp:773: While parsing node number 444 [GridSample -> "/0/decoder/decoder/layers.0/cross_attn/GridSample_output_0"]: ERROR: [TRT]: ModelImporter.cpp:774: --- Begin node --- ERROR: [TRT]: ModelImporter.cpp:775: input: "/0/decoder/decoder/layers.0/cross_attn/Reshape_5_output_0" input: "/0/decoder/decoder/layers.0/cross_attn/Reshape_6_output_0" output: "/0/decoder/decoder/layers.0/cross_attn/GridSample_output_0" name: "/0/decoder/decoder/layers.0/cross_attn/GridSample" op_type: "GridSample" attribute { name: "align_corners" i: 0 type: INT } attribute { name: "mode" s: "bilinear" type: STRING } attribute { name: "padding_mode" s: "zeros" type: STRING } ERROR: [TRT]: ModelImporter.cpp:776: --- End node --- ERROR: [TRT]: ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" ERROR: Failed to parse onnx file ERROR: failed to build network since parsing model errors.

[IronmanVsThanos]

Hi,

Has the RT-DETR been tested on the Jetson NX? When I try the a trained onnx model on the Jetson NX, the conversion failed with the below message. It works fine on the PC - converted and running. I am using Deepstream 6.0.

WARNING: [TRT]: onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. WARNING: [TRT]: onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped ERROR: [TRT]: ModelImporter.cpp:773: While parsing node number 444 [GridSample -> "/0/decoder/decoder/layers.0/cross_attn/GridSample_output_0"]: ERROR: [TRT]: ModelImporter.cpp:774: --- Begin node --- ERROR: [TRT]: ModelImporter.cpp:775: input: "/0/decoder/decoder/layers.0/cross_attn/Reshape_5_output_0" input: "/0/decoder/decoder/layers.0/cross_attn/Reshape_6_output_0" output: "/0/decoder/decoder/layers.0/cross_attn/GridSample_output_0" name: "/0/decoder/decoder/layers.0/cross_attn/GridSample" op_type: "GridSample" attribute { name: "align_corners" i: 0 type: INT } attribute { name: "mode" s: "bilinear" type: STRING } attribute { name: "padding_mode" s: "zeros" type: STRING } ERROR: [TRT]: ModelImporter.cpp:776: --- End node --- ERROR: [TRT]: ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" ERROR: Failed to parse onnx file ERROR: failed to build network since parsing model errors.

I have samae question with you

[IronmanVsThanos]

Hi,

Has the RT-DETR been tested on the Jetson NX? When I try the a trained onnx model on the Jetson NX, the conversion failed with the below message. It works fine on the PC - converted and running. I am using Deepstream 6.0.

WARNING: [TRT]: onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. WARNING: [TRT]: onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped ERROR: [TRT]: ModelImporter.cpp:773: While parsing node number 444 [GridSample -> "/0/decoder/decoder/layers.0/cross_attn/GridSample_output_0"]: ERROR: [TRT]: ModelImporter.cpp:774: --- Begin node --- ERROR: [TRT]: ModelImporter.cpp:775: input: "/0/decoder/decoder/layers.0/cross_attn/Reshape_5_output_0" input: "/0/decoder/decoder/layers.0/cross_attn/Reshape_6_output_0" output: "/0/decoder/decoder/layers.0/cross_attn/GridSample_output_0" name: "/0/decoder/decoder/layers.0/cross_attn/GridSample" op_type: "GridSample" attribute { name: "align_corners" i: 0 type: INT } attribute { name: "mode" s: "bilinear" type: STRING } attribute { name: "padding_mode" s: "zeros" type: STRING } ERROR: [TRT]: ModelImporter.cpp:776: --- End node --- ERROR: [TRT]: ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" ERROR: Failed to parse onnx file ERROR: failed to build network since parsing model errors.

have you slove this problem?

[IronmanVsThanos]

https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/RTDETR.md

404 - page not found

[prashant-dn]

https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/RTDETR.md

404 - page not found

there are 3 different READMEs now.