how to evaluate symmetric objects？

[FANzhaoxin666]

hi！i have trained a model on bottle! However, i find that it seems that is no evaluation codes about computing iou for such kinds of symmetric objects as described in the paper. Could please share how to evaluate symmetric objects in detail?

[lzhang57]

Hi,

You can add the below code snippets to https://github.com/google-research-datasets/Objectron/blob/master/objectron/dataset/eval.py#L187 to evaluate symmetric objects.

It first rotates the ground truth box along its vertical axis from 0~360 degrees, pick the rotated box which has the largest IoU with the predicted box, and use that rotated box to compute the eval metrics instead of the original ground truth.

def _get_rotated_box(box_point_3d, angle):
  """Rotate a box along its vertical axis.
  Args:
    box: Input box.
    angle: Rotation angle in rad.
  Returns:
    A rotated box
  """
  CENTER = 0
  BACK_TOP_LEFT = 3
  BACK_BOTTOM_LEFT = 1
  up_vector = box_point_3d[BACK_TOP_LEFT] - box_point_3d[BACK_BOTTOM_LEFT]
  rot_vec = angle * up_vector / np.linalg.norm(up_vector)
  rotation = rotation_util.from_rotvec(rot_vec).as_dcm()
  box_center = box_point_3d[CENTER]
  box_point_3d_rotated = np.matmul((box_point_3d - box_center), rotation) + box_center
  return box_point_3d_rotated
 
def evaluate_3d(self, box_point_3d, instance, N=100):
  """Evaluates a box in 3D.
  It computes metrics of view angle and 3D IoU.
  Args:
    box: A predicted box.
    instance: A 9*3 array of an annotated box, in metric level.
  Returns:
    The 3D IoU (float)
  """
  result = (None, None, 0.0)
  for angle in np.linspace(0, np.pi * 2, N):
    box_point_3d_rotated = self._get_rotated_box(box_point_3d, angle)
    azimuth_error, polar_error = self.evaluate_viewpoint(box_point_3d_rotated,
                                                         instance)
    iou = self.evaluate_iou(box_point_3d_rotated, instance)
    if iou > result[-1]:
      result = (azimuth_error, polar_error, iou)
  return result

[FANzhaoxin666]

Hi,

You can add the below code snippets to https://github.com/google-research-datasets/Objectron/blob/master/objectron/dataset/eval.py#L187 to evaluate symmetric objects.

It first rotates the ground truth box along its vertical axis from 0~360 degrees, pick the rotated box which has the largest IoU with the predicted box, and use that rotated box to compute the eval metrics instead of the original ground truth.

def _get_rotated_box(box_point_3d, angle):
  """Rotate a box along its vertical axis.
  Args:
    box: Input box.
    angle: Rotation angle in rad.
  Returns:
    A rotated box
  """
  CENTER = 0
  BACK_TOP_LEFT = 3
  BACK_BOTTOM_LEFT = 1
  up_vector = box_point_3d[BACK_TOP_LEFT] - box_point_3d[BACK_BOTTOM_LEFT]
  rot_vec = angle * up_vector / np.linalg.norm(up_vector)
  rotation = rotation_util.from_rotvec(rot_vec).as_dcm()
  box_center = box_point_3d[CENTER]
  box_point_3d_rotated = np.matmul((box_point_3d - box_center), rotation) + box_center
  return box_point_3d_rotated
 
def evaluate_3d(self, box_point_3d, instance, N=100):
  """Evaluates a box in 3D.
  It computes metrics of view angle and 3D IoU.
  Args:
    box: A predicted box.
    instance: A 9*3 array of an annotated box, in metric level.
  Returns:
    The 3D IoU (float)
  """
  result = (None, None, 0.0)
  for angle in np.linspace(0, np.pi * 2, N):
    box_point_3d_rotated = self._get_rotated_box(box_point_3d, angle)
    azimuth_error, polar_error = self.evaluate_viewpoint(box_point_3d_rotated,
                                                         instance)
    iou = self.evaluate_iou(box_point_3d_rotated, instance)
    if iou > result[-1]:
      result = (azimuth_error, polar_error, iou)
  return result

thanks