ros_deep_learning
ros_deep_learning copied to clipboard
`segnet` Node Crashes with "double free or corruption" Error During Launch
When trying to launch the segnet
node using the ros_deep_learning
package with the following command:
roslaunch ros_deep_learning segnet.ros1.launch input:=v4l2:///dev/video4 output:=display://0
The node crashes with the following error:
...
[ INFO] [1724721245.224459610]: allocated CUDA memory for 1280x720 image conversion
double free or corruption (!prev)
[segnet-3] process has died [pid 176, exit code -6, cmd /ros_deep_learning/devel/lib/ros_deep_learning/segnet /segnet/image_in:=/video_source/raw __name:=segnet __log:=/root/.ros/log/a3c743d6-6411-11ef-87d9-28d0431cfa8d/segnet-3.log].
log file: /root/.ros/log/a3c743d6-6411-11ef-87d9-28d0431cfa8d/segnet-3*.log
Full Log Output:
... logging to /root/.ros/log/a3c743d6-6411-11ef-87d9-28d0431cfa8d/roslaunch-ubuntu-140.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://ubuntu:39177/
SUMMARY
========
PARAMETERS
* /rosdistro: noetic
* /rosversion: 1.16.0
* /segnet/class_colors_path:
* /segnet/class_labels_path:
* /segnet/input_blob:
* /segnet/mask_filter: linear
* /segnet/model_name: fcn-resnet18-mhp-...
* /segnet/model_path:
* /segnet/output_blob:
* /segnet/overlay_alpha: 180.0
* /segnet/overlay_filter: linear
* /segnet/prototxt_path:
* /video_output/bitrate: 0
* /video_output/codec: unknown
* /video_output/resource: display://0
* /video_source/height: 0
* /video_source/loop: 0
* /video_source/resource: v4l2:///dev/video4
* /video_source/width: 0
NODES
/
segnet (ros_deep_learning/segnet)
video_output (ros_deep_learning/video_output)
video_source (ros_deep_learning/video_source)
auto-starting new master
process[master]: started with pid [153]
ROS_MASTER_URI=http://localhost:11311
setting /run_id to a3c743d6-6411-11ef-87d9-28d0431cfa8d
process[rosout-1]: started with pid [168]
started core service [/rosout]
process[video_source-2]: started with pid [175]
process[segnet-3]: started with pid [176]
process[video_output-4]: started with pid [177]
[ INFO] [1724721240.865774799]: opening video source: v4l2:///dev/video4
[gstreamer] initialized gstreamer, version 1.16.3.0
[gstreamer] gstCamera -- attempting to create device v4l2:///dev/video4
[ INFO] [1724721240.899158727]: opening video output: display://0
[OpenGL] glDisplay -- X screen 0 resolution: 1920x1080
[OpenGL] glDisplay -- X window resolution: 1920x1080
[gstreamer] gstCamera -- found v4l2 device: Intel(R) RealSense(TM) Depth Ca
[gstreamer] v4l2-proplist, device.path=(string)/dev/video4, udev-probed=(boolean)false, device.api=(string)v4l2, v4l2.device.driver=(string)uvcvideo, v4l2.device.card=(string)"Intel\(R\)\ RealSense\(TM\)\ Depth\ Ca", v4l2.device.bus_info=(string)usb-3610000.xhci-1.1, v4l2.device.version=(uint)330360, v4l2.device.capabilities=(uint)2225078273, v4l2.device.device_caps=(uint)69206017;
[gstreamer] gstCamera -- found 7 caps for v4l2 device /dev/video4
[gstreamer] [0] video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)800, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 15/1, 10/1, 5/1 };
[gstreamer] [1] video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 15/1, 10/1, 5/1 };
[gstreamer] [2] video/x-raw, format=(string)YUY2, width=(int)848, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 15/1, 5/1 };
[gstreamer] [3] video/x-raw, format=(string)YUY2, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 15/1, 5/1 };
[gstreamer] [4] video/x-raw, format=(string)YUY2, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 90/1, 60/1, 30/1, 15/1, 5/1 };
[gstreamer] [5] video/x-raw, format=(string)YUY2, width=(int)480, height=(int)270, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 90/1, 60/1, 30/1, 15/1, 5/1 };
[gstreamer] [6] video/x-raw, format=(string)YUY2, width=(int)424, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 90/1, 60/1, 30/1, 15/1, 5/1 };
[gstreamer] gstCamera -- selected device profile: codec=raw format=yuyv width=1280 height=720
[gstreamer] gstCamera pipeline string:
[gstreamer] v4l2src device=/dev/video4 do-timestamp=true ! video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720 ! appsink name=mysink sync=false
[gstreamer] gstCamera successfully created device v4l2:///dev/video4
[video] created gstCamera from v4l2:///dev/video4
------------------------------------------------
gstCamera video options:
------------------------------------------------
-- URI: v4l2:///dev/video4
- protocol: v4l2
- location: /dev/video4
- port: 4
-- deviceType: default
-- ioType: input
-- codec: raw
-- codecType: cpu
-- width: 1280
-- height: 720
-- frameRate: 30
-- numBuffers: 4
-- zeroCopy: true
-- flipMethod: none
-- loop: 0
------------------------------------------------
[gstreamer] opening gstCamera for streaming, transitioning pipeline to GST_STATE_PLAYING
[gstreamer] gstreamer changed state from NULL to READY ==> mysink
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter0
[gstreamer] gstreamer changed state from NULL to READY ==> v4l2src0
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter0
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> v4l2src0
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline0
[gstreamer] gstreamer message new-clock ==> pipeline0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> v4l2src0
segNet -- loading segmentation network model from:
-- prototxt:
-- model: networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx
-- labels: networks/FCN-ResNet18-MHP-512x320/classes.txt
-- colors: networks/FCN-ResNet18-MHP-512x320/colors.txt
-- input_blob 'input_0'
-- output_blob 'output_0'
-- batch_size 1
[TRT] TensorRT version 8.5.2
[TRT] loading NVIDIA plugins...
[TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[TRT] Registered plugin creator - ::Clip_TRT version 1
[TRT] Registered plugin creator - ::CoordConvAC version 1
[TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[TRT] Registered plugin creator - ::CropAndResize version 1
[TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[TRT] Could not register plugin creator - ::FlattenConcat_TRT version 1
[TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[TRT] Registered plugin creator - ::GroupNorm version 1
[TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2
[TRT] Registered plugin creator - ::LayerNorm version 1
[TRT] Registered plugin creator - ::LReLU_TRT version 1
[TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[TRT] Registered plugin creator - ::NMS_TRT version 1
[TRT] Registered plugin creator - ::Normalize_TRT version 1
[TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[TRT] Registered plugin creator - ::PriorBox_TRT version 1
[TRT] Registered plugin creator - ::ProposalDynamic version 1
[TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[TRT] Registered plugin creator - ::Proposal version 1
[TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TRT] Registered plugin creator - ::Region_TRT version 1
[TRT] Registered plugin creator - ::Reorg_TRT version 1
[TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[TRT] Registered plugin creator - ::RPROI_TRT version 1
[TRT] Registered plugin creator - ::ScatterND version 1
[TRT] Registered plugin creator - ::SeqLen2Spatial version 1
[TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[TRT] Registered plugin creator - ::SplitGeLU version 1
[TRT] Registered plugin creator - ::Split version 1
[TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[TRT] detected model format - ONNX (extension '.onnx')
[TRT] desired precision specified for GPU: FASTEST
[TRT] requested fasted precision for device GPU without providing valid calibrator, disabling INT8
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer message stream-start ==> pipeline0
[OpenGL] glDisplay -- display device initialized (1920x1080)
[video] created glDisplay from display://0
------------------------------------------------
glDisplay video options:
------------------------------------------------
-- URI: display://0
- protocol: display
- location: 0
-- deviceType: display
-- ioType: input
-- width: 1920
-- height: 1080
-- frameRate: 0
-- numBuffers: 4
-- zeroCopy: true
-- flipMethod: none
-- loop: 0
------------------------------------------------
[ INFO] [1724721241.062804675]: video_output node initialized, waiting for messages
[gstreamer] gstCamera -- onPreroll
[gstreamer] gstBufferManager recieve caps: video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, interlace-mode=(string)progressive
[gstreamer] gstBufferManager -- recieved first frame, codec=raw format=yuyv width=1280 height=720 size=1843200
[TRT] [MemUsageChange] Init CUDA: CPU +222, GPU +0, now: CPU 259, GPU 3991 (MiB)
[TRT] Trying to load shared library libnvinfer_builder_resource.so.8.5.2
[TRT] Loaded shared library libnvinfer_builder_resource.so.8.5.2
[cuda] allocated 4 ring buffers (1843200 bytes each, 7372800 bytes total)
[cuda] allocated 4 ring buffers (8 bytes each, 32 bytes total)
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysink
[gstreamer] gstreamer message async-done ==> pipeline0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysink
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline0
[cuda] allocated 4 ring buffers (2764800 bytes each, 11059200 bytes total)
[ INFO] [1724721241.876176221]: allocated CUDA memory for 1280x720 image conversion
[gstreamer] gstreamer message qos ==> v4l2src0
[TRT] [MemUsageChange] Init builder kernel library: CPU +302, GPU +390, now: CPU 583, GPU 4405 (MiB)
[TRT] native precisions detected for GPU: FP32, FP16, INT8
[TRT] selecting fastest native precision for GPU: FP16
[TRT] found engine cache file /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx.1.1.8502.GPU.FP16.engine
[TRT] found model checksum /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx.sha256sum
[TRT] echo "$(cat /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx.sha256sum) /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx" | sha256sum --check --status
[TRT] model matched checksum /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx.sha256sum
[TRT] loading network plan from engine cache... /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx.1.1.8502.GPU.FP16.engine
[TRT] device GPU, loaded /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx
[TRT] Loaded engine size: 22 MiB
[TRT] Deserialization required 27876 microseconds.
[TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +22, now: CPU 0, GPU 22 (MiB)
[TRT] Total per-runner device persistent memory is 1024
[TRT] Total per-runner host persistent memory is 68096
[TRT] Allocated activation device memory of size 7864320
[TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +7, now: CPU 0, GPU 29 (MiB)
[TRT] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[TRT]
[TRT] CUDA engine context initialized on device GPU:
[TRT] -- layers 25
[TRT] -- maxBatchSize 1
[TRT] -- deviceMemory 7864320
[TRT] -- bindings 2
[TRT] binding 0
-- index 0
-- name 'input_0'
-- type FP32
-- in/out INPUT
-- # dims 4
-- dim #0 1
-- dim #1 3
-- dim #2 320
-- dim #3 512
[TRT] binding 1
-- index 1
-- name 'output_0'
-- type FP32
-- in/out OUTPUT
-- # dims 4
-- dim #0 1
-- dim #1 21
-- dim #2 10
-- dim #3 16
[TRT]
[TRT] binding to input 0 input_0 binding index: 0
[TRT] binding to input 0 input_0 dims (b=1 c=3 h=320 w=512) size=1966080
[TRT] binding to output 0 output_0 binding index: 1
[TRT] binding to output 0 output_0 dims (b=1 c=21 h=10 w=16) size=13440
[TRT]
[TRT] device GPU, /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx initialized.
[TRT] segNet outputs -- s_w 16 s_h 10 s_c 21
[TRT] loaded 21 class labels
[TRT] class 00 color 0.000000 0.000000 0.000000 255.000000
[TRT] class 01 color 139.000000 69.000000 19.000000 255.000000
[TRT] class 02 color 222.000000 184.000000 135.000000 255.000000
[TRT] class 03 color 210.000000 105.000000 30.000000 255.000000
[TRT] class 04 color 255.000000 255.000000 0.000000 255.000000
[TRT] class 05 color 255.000000 165.000000 0.000000 255.000000
[TRT] class 06 color 0.000000 255.000000 0.000000 255.000000
[TRT] class 07 color 60.000000 179.000000 113.000000 255.000000
[TRT] class 08 color 107.000000 142.000000 35.000000 255.000000
[TRT] class 09 color 255.000000 0.000000 0.000000 255.000000
[TRT] class 10 color 245.000000 222.000000 179.000000 255.000000
[TRT] class 11 color 0.000000 0.000000 255.000000 255.000000
[TRT] class 12 color 0.000000 255.000000 255.000000 255.000000
[TRT] class 13 color 238.000000 130.000000 238.000000 255.000000
[TRT] class 14 color 128.000000 0.000000 128.000000 255.000000
[TRT] class 15 color 255.000000 0.000000 0.000000 255.000000
[TRT] class 16 color 255.000000 0.000000 255.000000 255.000000
[TRT] class 17 color 128.000000 128.000000 128.000000 255.000000
[TRT] class 18 color 192.000000 192.000000 192.000000 255.000000
[TRT] class 19 color 128.000000 128.000000 128.000000 255.000000
[TRT] class 20 color 128.000000 128.000000 128.000000 255.000000
[TRT] loaded 21 class colors
[ INFO] [1724721244.917549168]: model hash => 1987947487477878254
[ INFO] [1724721244.920395247]: hash string => /usr/local/bin/networks/FCN-ResNet18-MHP-512x320/fcn_resnet18.onnx/usr/local/bin/networks/FCN-ResNet18-MHP-512x320/classes.txt
[ INFO] [1724721244.923732964]: node namespace => /segnet
[ INFO] [1724721244.923812487]: class labels => /segnet/class_labels_1987947487477878254
[ INFO] [1724721244.943880455]: segnet node initialized, waiting for messages
[ INFO] [1724721245.192920619]: allocated CUDA memory for 1280x720 image conversion
[ INFO] [1724721245.224459610]: allocated CUDA memory for 1280x720 image conversion
double free or corruption (!prev)
[segnet-3] process has died [pid 176, exit code -6, cmd /ros_deep_learning/devel/lib/ros_deep_learning/segnet /segnet/image_in:=/video_source/raw __name:=segnet __log:=/root/.ros/log/a3c743d6-6411-11ef-87d9-28d0431cfa8d/segnet-3.log].
log file: /root/.ros/log/a3c743d6-6411-11ef-87d9-28d0431cfa8d/segnet-3*.log