Inference video does not have an inference box

[JasonChao-yue]

When deploying YOLOv5, the deployment was successful, but the target was not identified in the video during video inference. Is it a configuration error that caused the displayed video to not be boxed

[marcoslucianops]

Are there any bbox on the detection? Are you testing with the deepstream-app?

[JasonChao-yue]

Are there any bbox on the detection? Are you testing with the deepstream-app?

Thank you for your reply! I used deepstream-app. There was no bbox when I detected it, but the gpu occupancy rate was very high, indicating that I was reasoning. At the same time, I also noticed that I changed the engine, but it was all the same phenomenon.

[marcoslucianops]

Can you send the output from the terminal when you run the DeepStream?

[ghost]

Hi @marcoslucianops. I can confirm that I'm facing the same issue for YOLOv8. The model inference is happening but there are no bounding boxes (detections) drawn on the output in Deepstream 6.0.1. Strangely, 6.3 seems to work fine, though at this moment due to driver version restrictions. I need DS 6.0.1. Any leads would be appreciated.

[JasonChao-yue]

Can you send the output from the terminal when you run the DeepStream? I can send the output from the terminal when run the DeepStream The environment I am using is: Xavier nx ubuntu18.04 Deepstream 6.0.1 Yolov5 The terminal output result is： (ceshi) nvidia@nvidia-desktop:~/Downloads/DeepStream-Yolo-master$ deepstream-app -c deepstream_app_config.txt

Using winsys: x11 WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors. p0:00:04.446720746 21283 0x24f7a630 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/home/nvidia/Downloads/DeepStream-Yolo-master/model_b1_gpu0_fp32.engine INFO: [Implicit Engine Info]: layers num: 5 0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT onnx::Sigmoid_347 3x80x80x85
2 OUTPUT kFLOAT onnx::Sigmoid_404 3x40x40x85
3 OUTPUT kFLOAT onnx::Sigmoid_460 3x20x20x85
4 OUTPUT kFLOAT output 25200x85

0:00:04.447148140 21283 0x24f7a630 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /home/nvidia/Downloads/DeepStream-Yolo-master/model_b1_gpu0_fp32.engine 0:00:04.744482165 21283 0x24f7a630 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nvidia/Downloads/DeepStream-Yolo-master/config_infer_primary_yoloV5.txt sucessfully

Runtime commands: h: Print this help q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source. To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:194>: Pipeline ready

Opening in BLOCKING MODE NvMMLiteOpen : Block : BlockType = 261 NVMEDIA: Reading vendor.tegra.display-size : status: 6 NvMMLiteBlockCreate : Block : BlockType = 261 ** INFO: <bus_callback:180>: Pipeline running

**PERF: FPS 0 (Avg) **PERF: 25.61 (24.62) ** INFO: <bus_callback:187>: Pipeline paused

r ** INFO: <bus_callback:180>: Pipeline running

** INFO: <bus_callback:217>: Received EOS. Exiting ...

Quitting App run successful

[marcoslucianops]

You are export the ONNX model in wrong way. Please follow: https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/YOLOv5.md

The output of your model is incorrect.

[JasonChao-yue]

You are export the ONNX model in wrong way. Please follow: https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/YOLOv5.md

The output of your model is incorrect.

I did export it through the export method in the document and it has been displayed successfully. The command used is: python3 export_ YOLOV5. py - w YOLOv5s. pt -- dynamic, which location did you use to determine the error in exporting ONNX?

[JasonChao-yue]

You are export the ONNX model in wrong way. Please follow: https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/YOLOv5.md

The output of your model is incorrect. @marcoslucianops @linga-abhishek

The problem has been resolved. It is indeed a problem with the generated ONNX model. The problem occurred when using the YOLOv5 5.0 environment to generate ONNX, but was resolved by changing it to YOLOv5 6.0

[HeeebsInc]

@JasonChao-yue how did you find what version you needed? I am running into the same issue - is there a PyTorch method I can use to see what version I trained the model with? This is the first time that a version is causing issues, onnx + deepstream 6.2 works but does not work for 6.3 with yolov5

[marcoslucianops]

Use the master branch to convert all YOLOv5 versions.

[HeeebsInc]

I'll try that out. Thanks Marco!

[HeeebsInc]

@marcoslucianops i switched from v7 back to master but I am still not getting detections with a custom yolov5 model. The onnx export and engine build is successful, but during inference there are no detections at all (almost like everything is suppressed).

[marcoslucianops]

@HeeebsInc can you send the output from the terminal?

[HeeebsInc]

@marcoslucianops

WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: Deserialize engine failed because file path: best_ap-1-fp16.engine open error
0:00:03.809266374    54      0x4445af0 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1976> [UID = 1]: deserialize engine from file : best_ap-1-fp16.engine failed
0:00:03.872457154    54      0x4445af0 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2081> [UID = 1]: deserialize backend context from engine from file :best_ap-1-fp16.engine failed, try rebuild
0:00:03.873130954    54      0x4445af0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2002> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [TRT]: onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: Tensor DataType is determined at build time for tensors not marked as input or output.

Building the TensorRT Engine

Building complete

0:02:14.251159294    54      0x4445af0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2034> [UID = 1]: serialize cuda engine to file: /ze/data/Experiments/TEST/TE-563/specs/186-best_ap-1-fp16.engine successfully
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT input           3x544x960       
1   OUTPUT kFLOAT boxes           32130x4         
2   OUTPUT kFLOAT scores          32130x1         
3   OUTPUT kFLOAT classes         32130x1 

steps for what i did

1. python3 python3 export_yoloV5.py -w {self.model_path} -s {self.height} {self.width}
2. pgie

[property]
gpu-id = 0
model-color-format = 0
labelfile-path = labels.txt
process-mode = 1
num-detected-classes = 3
interval = 0
batch-size = 1
gie-unique-id = 1
maintain-aspect-ratio = 0
network-mode = 2
workspace-size = 6500
cluster-mode = 2
network-type = 0
force-implicit-batch-dim = 1
infer-dims = 3;544;960
net-scale-factor = 0.0039215697906911373
onnx-file = best_ap.onnx
model-engine-file = best_ap-1-fp16.engine
parse-bbox-func-name = NvDsInferParseYolo
custom-lib-path = /opt/nvidia/deepstream/deepstream-6.3/sources/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name = NvDsInferYoloCudaEngineGet
symmetric-padding = 0

[class-attrs-0]
post-cluster-threshold = 0.1
detected-min-w = 1
detected-min-h = 1
nms-iou-threshold = 0.1
topk = 1000
pre-cluster-threshold = 0.1

[class-attrs-1]
post-cluster-threshold = 0.1
detected-min-w = 1
detected-min-h = 1
nms-iou-threshold = 0.1
topk = 1000
pre-cluster-threshold = 0.1

[class-attrs-2]
post-cluster-threshold = 0.1

[marcoslucianops]

@HeeebsInc the model output seems correct. Did you train you model in 960x544 size?

[HeeebsInc]

@marcoslucianops After running another test I believe there may be an issue with the new deepstream 6.3 implementation. Below is a config that I used for both 6.2 and 6.3 (the only difference being the path of custom-lib-path=/opt/nvidia/deepstream/deepstream-6.3/sources/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so

When running through deepstream 6.2, the model gets detections while nothing it detected in deepstream 6.2. They both use the same .onnx file but a different .engine file.

[marcoslucianops]

I will do some tests later today. Can you send me your model in the email?

[marcoslucianops]

Are you using Jetson or x86 platform?

[HeeebsInc]

x86 - 3090 with Driver Version 525.125.06 + Cuda 12.1

I cannot send the exact model but I can prepare one that should be identical. It will likely be next week though.

I think a good first attempt would be to export a yolov5l model at 544 960 python3 export.py -s 544 960 --weights yolov5l.pt and that should show the same behavior. I will try that on my end as well.

[marcoslucianops]

The export.py from YOLOv5 repo doesn't work with the DeepStream-Yolo, You should use the export_yoloV5.py from DeepStream-Yolo/utils. Are you training your model with image normalization in the pre-processing?

[HeeebsInc]

The export.py from YOLOv5 repo doesn't work with the DeepStream-Yolo, You should use the export_yoloV5.py from DeepStream-Yolo/utils. Are you training your model with image normalization in the pre-processing?

Apologies that was a typo on my end. I am using export_yolov5.py from your repo (not ultralytics). I will confirm our normalization technique. I believe we did not change anything from the traditional ultralytics implementation when it comes to pre processing.

[marcoslucianops]

Got it. The ultralytics implementation doesn't uses normalization (mean and standard deviation values of [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225] for example).

[marcoslucianops]

If you are using normalization, need to change some parameters in the config_infer file to run the inference.

[HeeebsInc]

@marcoslucianops confirmed with our team that we have not messed with normalization, so it's the same as default ultralytics. We're you able to reproduce the issue?

[marcoslucianops]

What's your YOLOv5 version?

[marcoslucianops]

If possible, share the steps to reproduce this issue (with YOLOv5 version and which branch of YOLOv5 repo you are using).

[HeeebsInc]

1. Pull master branch of this repository
2. pull yolov5l.pt weights
3. Pull master branch of ultralytics, and run pip install onnx onnxruntime
4. convert yolov5l.pt weights to yolov5.onnx, with a batch size of 1 and input resolution of 544x960 (hxw)
5. use the config I dropped above, modifying the paths to what you generated above

[marcoslucianops]

DeepStream 6.3

[HeeebsInc]

@marcoslucianops thanks for checking. Did you use the same pgie.txt config I dropped above? If not can you drop the config you used?

Again, really appreciate your help with this.

[marcoslucianops]

Yes, it's the same, I just changed the paths, the network-mode to 0, and workspace-size to 2000 (due to my GPU limitation). But I recommend you use the config_infer_primary_yoloV5.txt file as sample because there are some parameters like maintain-aspect-ratio and symmetric-padding that are used in the YOLOv5, and the correct nms-iou-threshold placement (in the [class-attrs-all] property).