Callback for Plotting Bounding Boxes, Segmentation Masks.

[oke-aditya]

🚀 Feature

The next release of torchvision 0.9.0 will pack a few plotting utilities that work with detection and segmentation models. E.g. draw_bounding_boxes, draw_segmentation_masks,

We will have merged detection models from torchvision, (FRCNN and RetinaNet (soon)) into bolts. So this can be a nice time to implement a callback.

Motivation

It would be really nice to a Callback that allows the users to log their predictions to TensorBoard, etc.

Pitch

I'm not sure how callbacks work (will dive into it) but something like confused logit callback can work I guess (thanks @akihironitta).

Detection models in their validation mode output the predicted boxes, which are really good to visualize. Maybe we can do something on_validation_step_end?

Alternatives

I'm not sure if we would leave it to the user for such implementation, but I think it is nice to have.

Additional context

~Wait till next release fo torchvision, these features are still not released~ Both the features are released since torchvision 0.9 they should be available.

[Programmer-RD-AI]

hi,

I wanted to know why is the draw_bounding_boxes

not using cv2

import cv2 xmin,ymin,xmax,ymax = 317,265,556,342 x = xmin y = ymin w = xmax - xmin h = ymax - ymin crop = img[y:y+h,x:x+w] plt.imshow(crop) plt.imshow(cv2.rectangle(img,(x,y),(x+w,y+h),(200,0,0),2))

Example notebook https://github.com/Programmer-RD-AI/Microcontroller-Detection-Learning-Detectron2/blob/main/00.ipynb

I just wanted to know I am trying to do this issue.

With best regards, Ranuga

[oke-aditya]

Hi @Programmer-RD-AI draw_bounding_boxes and draw_segmentation_masks or in fact all the utilities written in torchvision.utils do not use cv2

The major reason is to avoid an extra opencv dependency. PIL works well out of box, which is a needed dependency for torchvision. Also the utils API works really well with the models provided in torchvision.models

A simple example for them is available over torchvision gallery.

https://pytorch.org/vision/stable/auto_examples/plot_visualization_utils.html#sphx-glr-auto-examples-plot-visualization-utils-py

[oke-aditya]

Your above code can be rewritten very easily with torchvision utils

import torch
from torchvision.io import read_image

img = read_image("pathto_file.jpg")

xmin, ymin, xmax, ymax = 317, 265, 556, 342
bbox_tensor = torch.tensor([xmin, ymin, xmax, ymax], dtype=torch.float)

res = draw_bounding_boxes(img, bbox_tensor, colors="red", label=["Image"], width=3)

draw_bounding_boxes is more robust, it can directly draw boxes on tensor, add a label, fill with same semi transparent color (use fill=True). You can also adjust the width of the box.

Read more about utilities in the docs.

https://pytorch.org/vision/stable/utils.html#torchvision.utils.draw_bounding_boxes

[Programmer-RD-AI]

ok thank you @oke-aditya

I will check the resources.

With best regards, Ranuga

[roboserg]

Curious about the progress on this? Is there a built-in way in PyTorch Lighting, or Flash or Bolts to visualize bboxes from model predictions?