Does evaluation API support OBB

[lfurtadoh]

Search before asking

• [X] I have searched the Supervision issues and found no similar feature requests.

Question

Hello! I am working with YOLOv8 for an object detection issue. I need to evaluate my model on a testset with a custom filtering method, and would like to be able to use supervision to do things like calculate mAP50 and create a confusion matrix, with the MeanAveragePrecision and ConfusionMatrix classes. I saw in issue #1227 that support for Yolo OBB dataset format is getting added.

If i do this:

dataset = sv.DetectionDataset.from_yolo(... is_obb = True)

model = YOLO(...)
    def callback(image: np.ndarray) -> sv.Detections:
    result = model(image)[0]
    return sv.Detections.from_ultralytics(result)

mean_average_precision = sv.MeanAveragePrecision.benchmark(
    dataset = dataset,
    callback = callback
)

Will it work properly, considering the dataset and detections are OBB's? Or is it still something to be added?

Additional

No response

[LinasKo]

Hi @lfurtadoh 👋

Looking at the code, MeanAveragePrecision does not support OBB - only standard object detection boxes. Given we're expanding our OBB support, I'll open a feature request for the community.

If you're willing, feel free to help us out!

[LinasKo]

I looked into it deeper. In my opinion, the cleanest solution uses shapely which we don't currently have as a dependency.

@SkalskiP, let's speak about that next time we meet.

[Bhavay-2001]

Hi @SkalskiP, is the team considering this issue? I have started to think about where can the changes possibly be.

[Bhavay-2001]

Hi @LinasKo, any updates on this?

[onuralpszr]

@Bhavay-2001 Linas is in vacation so It would be best to ask either Piotr or wait for them to response :)

[SkalskiP]

@onuralpszr, thanks a lot for keeping the hand on the pulse 🙏🏻 as for OBB support, I'm happy to expand support but preferably without shapely as a dependency.

However, before we mAP, we need to calculate IoU for OBBs and solve problems with InferenceSlicer for OBBs.

[Bhavay-2001]

Should we then open the issue to calculate IoU for OBBs? I have worked with OBBs so maybe I can help there

[SkalskiP]

@Bhavay-2001, I think we can work on this here.

[Bhavay-2001]

I saw the code which calculates iou for normal boxes[xmin, xmax,...]. The code is here. I think considering this as baseline, we can change this function according to OBB boxes. Is this in the right direction?

[SkalskiP]

Hi @Bhavay-2001 👋🏻 definitely not change. We should create a new one, oriented_box_iou_batch, that implements desired logic.

[Bhavay-2001]

Yup sorry, I meant adding a new function. Should I try this one then? If yes, please review my pending PRs so that I can complete them and work on this one.

[patel-zeel]

Hi @SkalskiP, I am surprised by the fact that research on OBB models is increasing in top conferences but no library I am aware of consistently provides low-level functions to probe into the results for researchers. We have been using supervision for our axis-aligned bounding boxes pipeline, and it has been an amazing experience. Thanks and cudos to you and all the contributors for making such a stunning library!

This motivates me to try implementing the IoU function for OBB, followed by helping you with other functionalities to make the OBB evaluation pipeline as smooth as possible.

Since you have mentioned without shapely, shall we go ahead with the same logic as probiou logic from Ultralytics? I checked their code and the logic should work with pure numpy without additional dependencies.

[LinasKo]

Hi @patel-zeel 👋

You're asking the perfect question at the perfect time.

Starting from the next release (est. tomorrow), we shall begin an initiative to build up a suite of metrics in supervision. As part of this, we'd love to make OBBs first-class-citizens, just like masks and xyxy boxes are.

We'll definitely need oriented_box_iou_batch. However, while I know very little about algorithms for OBBs, the proposed probiou feels off. Do we need to model our boxes as a Gaussian distribution?

Is there no more common way to compute OBB IoU? If needed, we could model them as convex polygons.

[patel-zeel]

Thanks for the updates @LinasKo. I am happy to know about supervision's plan, especially prioritizing OBBs.

I was proposing probiou because it is actively being used by ultralytics but honestly, when I first knew about the Gaussian distribution point from one of my colleagues, I had the same question, "Do we really need Gaussians when dealing with OBBs"? I am laying down our options here as I perceive them:

• We adapt to what is implemented in widely used libraries such as ultralytics and mmrotate. I have to check yet for mmrotate.
• We do a survey of what is being used in the recent research papers published at A* conferences such as ICML, CVPR, ICCV, KDD (as per my current mental database, most of them use mmrotate ). We reason about the pros and cons of multiple methods and choose the best one which fits well in supervision.
• We design our own method to do this. Maybe it has something to do with basic geometry rules and we may be able to figure out some formulas in pure numpy leveraging angle information. Some points to consider here:
  • Two widely used methods to specify OBB are xywhr and xyxyxyxy. There should be a robust way to transfer from xyxyxyxy to xywhr because xyxyxyxy may not be perfectly rectangular depending on the objects and resolution of imagery used in labeling. ultralytics already has those functions (I haven't taken a deep look if supervision also has)
  • What we develop needs to have some validation because if there is a trivial way, there has to be a reason why other people are not using it.

I am all ears for your feedback, thoughts and plans for this and would happily contribute to make this happen.

[LinasKo]

Hi @patel-zeel, 👋

Let's do it in 2 steps. First, let's implement an approximation by viewing the obb as a polygon. In the file supervision/detection/utils.py there's a function polygon_to_mask. We can use that to convert it to a mask, and then call mask_iou_batch. All of this should live inside a new function oriented_box_iou_batch.

Let's implement that in the first PR, and then refine as needed.

Do you have a but of time to work on this, @patel-zeel?

[patel-zeel]

Sound good, @LinasKo. Sure, I'd like to work on this.

[LinasKo]

Assigned it to you. Let me know if you need any help!

[LinasKo]

The initial implementation for this is complete. Rather crudely, we treat the oriented box as a polygon, convert it to a mask (taking up cells in a grid), and then compute the IoU of that. For our purposes, it is sufficient, e.g. as a basis for metrics computation.

If anyone's interested, we'd welcome contributions which could make the calculations more precise. This would indeed involve surveying what's out there, as I don't know the best methods.

To address a points right away:

• @patel-zeel proposed the conversion to xywhr. We'd generally like to have more streamlined conversion methods of mask, box, etc types to commonly used formats and back, but haven't had the time to work on it ourselves. If a new format such as xywhr helps in the computation, we're happy to add converters.

[patel-zeel]

Hi @LinasKo and @SkalskiP, now that we have the oriented_box_iou_batch function, how do we plan to integrate it into the evaluation API?

I have a working version locally for MeanAveragePrecision with the following changes:

1. Remove the following check.

https://github.com/roboflow/supervision/blob/1d4b61f20cf8d152b936a8c64f7918b65f52bcc3/supervision/metrics/mean_average_precision.py#L39-L42

2. Add one more condition for MetricTarget.ORIENTED_BOUNDING_BOXES and use oriented_box_iou_batch function.

https://github.com/roboflow/supervision/blob/1d4b61f20cf8d152b936a8c64f7918b65f52bcc3/supervision/metrics/mean_average_precision.py#L171-L178

3. Replace np.ndarray with np.asarray.

https://github.com/roboflow/supervision/blob/1d4b61f20cf8d152b936a8c64f7918b65f52bcc3/supervision/metrics/mean_average_precision.py#L334-L336

[LinasKo]

Excellent, it seems like you're near the destination already. Feel free to open a PR when you're there!

I'm drawn to other matters and will be slightly less available for in-depth discussions this week. Ideally - let's chat in the PR review :)