darknet_ros
darknet_ros copied to clipboard
leggedrobotics
Sync boundingboxes with images
Hi. I am trying to sync boundingboxes with images and I am using message filter to do that. Before that I also modified the header of bbx using _imageHeader. Then I found out the message doesn't coordinate to each other between bbx and images. I used an image publisher working at 0.2Hz and it publishes two pictures in turns. I used rostopic echo /darknet_ros/bounding_boxes to check the message and got the output below.(Followed with an analysis of this sequence)
header:
seq: 57526
stamp:
secs: 1532613366
nsecs: 48888444
frame_id: "detection"
image_header:
seq: 30
stamp:
secs: 1532613366
nsecs: 48888444
frame_id: ''
bounding_boxes:
-
Class: "sheep"
probability: 0.986069023609
xmin: 148
ymin: 34
xmax: 251
ymax: 218
-
Class: "sheep"
probability: 0.98369383812
xmin: 75
ymin: 56
xmax: 154
ymax: 230
-
Class: "sheep"
probability: 0.92605394125
xmin: 95
ymin: 4
xmax: 162
ymax: 83
-
Class: "sheep"
probability: 0.719430565834
xmin: 2
ymin: 75
xmax: 88
ymax: 256
-
Class: "sheep"
probability: 0.55137616396
xmin: 0
ymin: 0
xmax: 33
ymax: 74
-
Class: "sheep"
probability: 0.543074071407
xmin: 154
ymin: 29
xmax: 249
ymax: 115
-
Class: "sheep"
probability: 0.569041490555
xmin: 113
ymin: 0
xmax: 154
ymax: 15
---
header:
seq: 57527
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: "detection"
image_header:
seq: 31
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: ''
bounding_boxes:
-
Class: "sheep"
probability: 0.986069023609
xmin: 148
ymin: 34
xmax: 251
ymax: 218
-
Class: "sheep"
probability: 0.98369383812
xmin: 75
ymin: 56
xmax: 154
ymax: 230
-
Class: "sheep"
probability: 0.92605394125
xmin: 95
ymin: 4
xmax: 162
ymax: 83
-
Class: "sheep"
probability: 0.719430565834
xmin: 2
ymin: 75
xmax: 88
ymax: 256
-
Class: "sheep"
probability: 0.55137616396
xmin: 0
ymin: 0
xmax: 33
ymax: 74
-
Class: "sheep"
probability: 0.543074071407
xmin: 154
ymin: 29
xmax: 249
ymax: 115
-
Class: "sheep"
probability: 0.569041490555
xmin: 113
ymin: 0
xmax: 154
ymax: 15
---
header:
seq: 57528
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: "detection"
image_header:
seq: 31
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: ''
bounding_boxes:
-
Class: "sheep"
probability: 0.986069023609
xmin: 148
ymin: 34
xmax: 251
ymax: 218
-
Class: "sheep"
probability: 0.98369383812
xmin: 75
ymin: 56
xmax: 154
ymax: 230
-
Class: "sheep"
probability: 0.92605394125
xmin: 95
ymin: 4
xmax: 162
ymax: 83
-
Class: "sheep"
probability: 0.719430565834
xmin: 2
ymin: 75
xmax: 88
ymax: 256
-
Class: "sheep"
probability: 0.55137616396
xmin: 0
ymin: 0
xmax: 33
ymax: 74
-
Class: "sheep"
probability: 0.543074071407
xmin: 154
ymin: 29
xmax: 249
ymax: 115
-
Class: "sheep"
probability: 0.569041490555
xmin: 113
ymin: 0
xmax: 154
ymax: 15
---
header:
seq: 57529
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: "detection"
image_header:
seq: 31
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: ''
bounding_boxes:
-
Class: "bicycle"
probability: 0.573687434196
xmin: 211
ymin: 86
xmax: 253
ymax: 118
-
Class: "car"
probability: 0.889243423939
xmin: 18
ymin: 57
xmax: 252
ymax: 200
---
header:
seq: 57530
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: "detection"
image_header:
seq: 31
stamp:
secs: 1532613371
nsecs: 51276922
frame_id: ''
bounding_boxes:
-
Class: "bicycle"
probability: 0.573687434196
xmin: 211
ymin: 86
xmax: 253
ymax: 118
-
Class: "car"
probability: 0.889243423939
xmin: 18
ymin: 57
xmax: 252
ymax: 200
Actually there are only two timestamps "1532613366.48888444" and "1532613371.51276922" with five frames("sheep*7", "sheep*7", "sheep*7", "bicycle, car", "bicycle, car"), as the node keeps publishing bbxs at some other frequency. The first timestamp corresponds with an image with sheep, whereas the second timestamp is about cars. In the middle of the sequence there are two frames (57527,57528) that the bbxs have the timestamp with "1532613371.51276922" but detecting "sheep", which can be seen as a wrong correspondency between image and bbx. I wonder why that will happen, and are there any ways to fix that? Thanks.
If any help, here is my image_publisher code.
#!/usr/bin/env python3
from __future__ import print_function
import roslib
import sys
import rospy
import cv2
import std_msgs.msg
from std_msgs.msg import String
from sensor_msgs.msg import Image
from cv_bridge import CvBridge, CvBridgeError
# load executable path
# referenced from https://answers.ros.org/question/236116/how-do-i-access-files-in-same-directory-as-executable-python-catkin/
import os, rospkg
rp = rospkg.RosPack()
pwd = rp.get_path("img_publisher")+"/"
def talker():
bridge=CvBridge()
pub = rospy.Publisher('image_src', Image, queue_size=10)
rospy.init_node('talker',anonymous=True)
rate = rospy.Rate(hz)
i=0
predefined_width=256
predefined_height=256
while not rospy.is_shutdown():
i=(i+1)%2
img=cv2.imread(pwd+"imgs/%d.jpg"%(i))
img=cv2.resize(img,(predefined_width,predefined_height),interpolation=cv2.INTER_CUBIC)
ros_img=bridge.cv2_to_imgmsg(img,"bgr8")
h = std_msgs.msg.Header()
h.stamp=rospy.Time.now()
rospy.loginfo("print img at %d. %d"%(h.stamp.secs,h.stamp.nsecs))
ros_img.header=h
pub.publish(ros_img)
rate.sleep()
if __name__ == '__main__':
try:
hz=0.2
talker()
except rospy.ROSInterruptException:
pass
@mengyuest Do you solve the problem or has any clue about this problem? I have the same issue here.
Yes, though tough for me at first. Notice that they have three buffers to handle things(cache image, detection and publishing) in different stages (like in a pipeline, for raising the throughput I guess). So you need to fetch data in correct index, or will in wrong order. My solution is to also set buffers for headers. When receiving an image, save its header to the corresponding index(same as the index of image buffers) of header buffers, and finally when publishing messages, extract the corresponding header from header buffers. Or you can try to remove all the threads and handle things in one strict sequence as an image callback function (I tried, but got efficiency lost).
@mengyuest Thanks a lot. Will work on it later. BTW: is it possible for you to share your darknet ros package?
@mengyuest @rpfly3 Hi. Potentially fixed. Only tested on my setup. Would appreciate other people testing it.
PR here: https://github.com/leggedrobotics/darknet_ros/pull/113 Branch for testing here: https://github.com/Texas-Aerial-Robotics/darknet_ros/tree/headerFixForUpsteam
You can always change the header that is being published in darknet_ros/src/YoloObjectDetector.cpp line 622. This might do the work: boundingBoxesResults_.image_header = headerBuff_[(buffIndex_ + 1) % 3]; //boundingBoxesResults_.header.stamp = ros::Time::now(); //line replaced by the next one boundingBoxesResults_.header.stamp = boundingBoxesResults_.image_header.stamp;
