[FR] Add support for rotated bounding boxes

[HamLam]

Proposal Summary

Could you please provide support for the visualization of rotated bounding boxes for coco dataset? The bounding box parameters for rotated boxes would include a fifth entry (rotated angle in radian) instead of the usual four for coco formatted dataset. Thank you!

Motivation

• What is the use case for this feature? Rotated bounding boxes are commonly bound on coco dataset in object detection.

• Why is this use case valuable to support for FiftyOne users in general? It is a valuable feature because it expands the capability of FiftyOne

What areas of FiftyOne does this feature affect?

• [ *] App: FiftyOne application
• [* ] Core: Core fiftyone Python library
• [ ] Server: FiftyOne server

Details

(Use this section to include any additional information about the feature. If you have a proposal for how to implement this feature, please include it here.)

Willingness to contribute

The FiftyOne Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• [ ] Yes. I can contribute this feature independently.
• [ ] Yes. I would be willing to contribute this feature with guidance from the FiftyOne community.
• [ *] No. I cannot contribute this feature at this time.

[ehofesmann]

I definitely agree that rotated bounding boxes would be valuable to have in FiftyOne. While it shouldn't be too much of a lift to add, it will require updates to the App to support rotated bbox visualization, and we also need to decide on how the rotation angle will be stored in the dataset. It seems like the best options would be either:

1. add the rotation angle into the bounding_box list allowing it to either be 4 or 5 elements in length
2. add a new dedicated attribute rotation_angle to the Detection

[radao]

Any plans to add rotated boxes in the near term?

[robmarkcole]

This is an increasingly common format for aerial datasets and its inclusion would benefit this entire domain

[brimoor]

If we were to add a dedicated rotated bounding box type, say RotatedDetection, my inclination would be to represent it like this:

class RotatedDetection(Detection):
    """A rotated detections.

    Args:
        label (None): the label string
        bounding_box (None): a list of relative bounding box coordinates in
            ``[0, 1]`` in the following format::

                [<x-center>, <y-center>, <width>, <height>]

        theta (None): the counter-clockwise rotation of the box, in radians
    """

[brimoor]

Note that one can already visualize this data by storing it as polylines.

Perhaps it would be sufficient to just add a Polyline.from_rotated_box() factory method?

import cv2
import numpy as np

import fiftyone as fo
import fiftyone.core.labels as fol


def from_rotated_box(xc, yc, w, h, theta, **kwargs):
    R = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
    x = 0.5 * w * np.array([1, -1, -1, 1])
    y = 0.5 * h * np.array([1, 1, -1, -1])
    points = R.dot(np.stack((x, y))).T + np.array((xc, yc))
    return fol.Polyline(points=[points.tolist()], closed=True, **kwargs)


polylines = []
for _ in range(10):
    xc, yc = 0.2 + 0.6 * np.random.rand(2)
    w, h = 1.5 * (min(xc, yc, 1 - xc, 1 - yc)) * np.random.rand(2)
    theta = 2 * np.pi * np.random.rand()
    polyline = from_rotated_box(xc, yc, w, h, theta, label="rotated box")
    polylines.append(polyline)


cv2.imwrite("/tmp/image.png", np.full((128, 128, 3), 255, dtype=np.uint8))

sample = fo.Sample(
    filepath="/tmp/image.png",
    boxes=fo.Polylines(polylines=polylines),
)

dataset = fo.Dataset()
dataset.add_sample(sample)

session = fo.launch_app(dataset)

[adriantre]

My first instinct is that we don't need a dedicated type for this. The user can already add an attribute 'rotation' to the Detection sample. Then, if it is present, rotate around the provided coordinate on preview.

Especially for aerial applications, storing the rotation as azimuth (clockwise degrees from north) would make sense. label-studio also returns it in this format.

One thing to note is that Detectron2 specifies RotatedBoxes as (x_center, y_center, width, height, angle), and Detections in Fiftyone are stored as [<top-left-x>, <top-left-y>, <width>, <height>]. It does not matter for applying the rotation, but for evaluation (e.g. iou) and exporting to and from other formats, this needs to be adapted.

[huynguyenxuan212]

it should be integrated with cvat backend too

[jamesljlster]

Hi,

I'm a Machine Learning Engineer, looking forward to rotated bounding box support from FiftyOne. I stand for a dedicated rotated bounding box type for the following reasons:

• Rotated bounding box is not just bounding box adding with theta.
  When theta of a rotated bounding box is changed, box size should be adjusted too, unless they are really correct.

• Sometimes rotated bounding box cannot be translated to ideal bounding box directly. The following two objects share the same rotated bounding box, but having different bounding boxes:
  Personally, I always treat rotated bounding box and bounding box as different annotations, because they cannot reinterpret to each other.

• It's always more convenient to check what it is through its type, rather than contents.

[Rephil2]

I don't know if this is useful to post here but here are my thoughts on a solution to my own situation with CVAT, which outputs a torch vision dataset with bboxes as [x1,y1,x2,y2] and clockwise rotation of degrees (r), so that is my starting point.

i had an issue with fo.Polyline.from_rotated_box as it seems to want the [xc, yc, w, h] all normalized before performing the rotation. I suppose it's possible to do it this way but the fifityone implementation seems to only create skewed boxes unless they are in 0 degrees angle (Or I was just too dumb to figure out how to handle this). So I made my own solution to the issue. its obviously not perfect but:

def rotate_bbox_and_normalize(x1, y1, x2, y2, rotation, image_width, image_height):
    """
    Rotates a bounding box and normalizes the coordinates to be between 0 and 1.
    Args:
        x1 (float): x coordinate of the top left corner of the bounding box
        y1 (float): y coordinate of the top left corner of the bounding box
        x2 (float): x coordinate of the bottom right corner of the bounding box
        y2 (float): y coordinate of the bottom right corner of the bounding box
        rotation (float): rotation of the bounding box in degrees
        image_width (int): width of the image
        image_height (int): height of the image
    Returns:
        list: list of normalized coordinates of the rotated bounding box
    """

    def rotate_point(point, angle, origin=(0, 0)):
        """
        Rotates a point counterclockwise by a given angle around a given origin.
        The angle should be given in radians.
        Args:
            point (tuple): coordinates of the point to be rotated
            angle (float): angle of rotation in radians
            origin (tuple): point to rotate around
        Returns:
            tuple: coordinates of the rotated point
        """
        x, y = point
        ox, oy = origin
        qx = ox + math.cos(angle) * (x - ox) - math.sin(angle) * (y - oy)
        qy = oy + math.sin(angle) * (x - ox) + math.cos(angle) * (y - oy)
        return qx, qy

    rotation_radians = rotation * math.pi / 180 # Convert to radians

    corners = [(x1, y1), (x2, y1), (x2, y2), (x1, y2)]
    # image_center = (image_width / 2, image_height / 2)
        # Rotation origin point
    bbox_center = ((x1 + x2) / 2, (y1 + y2) / 2)

    rotated_bbox = []
    for corner in corners:
        rotated_corner = rotate_point(corner, rotation_radians, origin=bbox_center)
        rotated_bbox.append(rotated_corner)
    
    normalized_rotated_bbox = []
    for corner in rotated_bbox:
        x_normalized = corner[0] / image_width
        y_normalized = corner[1] / image_height
        normalized_rotated_bbox.append([x_normalized, y_normalized])
    
    return [normalized_rotated_bbox]

after defining this you can then in your loop do something like this:

	polyline = fo.Polyline.from_rotated_box(
            # Set box coordinates and rotation to 0 as they will be updated after this line is created.
	    0,0,0,0,0,
	    label="cat",
	    mood="surly",  # custom attribute
	)
        # Call my hotfix mess of a custom function that spits out the relevant polyline points
	polyline['points'] = rotate_bbox_and_normalize(x1, y1, x2, y2, float(row['rotation']), image_width, image_height)
	# Create FiftyOne Polyline
	bbox_polylines.append(polyline)

# Store detections in a field name of your choice
sample["ground_truth"] = fo.Polylines(polylines=bbox_polylines)

The box is rotated around its center point [xc, yc].

[brimoor]

@Rephil2 good call, our from_rotated_box() method definitely needs to know the width/height of the image in order to correctly apply the rotation and then convert to normalized coordinates 👍