square elongation for cropmodel

[bw4sz]

From an unpublished paper, which I can later link.

Instead of cropping tightly around the object, we convert the bounding boxes into square shapes by extending the shorter side to match the length of the longer side. Another way to think of it is that we are doing padding with natural pixels. Our box expansion approach maintains the original aspect ratios of objects without introducing artifacts, reducing the likelihood of misidentification.

We could do this for the cropmodel workflow.

The relevant code is here:

https://github.com/weecology/DeepForest/blob/8815d4b89e6e482c46bff7ed7d574efb44052432/deepforest/model.py#L161

[ethanwhite]

This is a really cool idea!

[NaganathanM]

Hi! If my understanding is correct, you want to adjust the xmin, xmax or ymin, ymax depending on the longer side, to make the box square. Shall I take up and work on this if it's still open?

[bw4sz]

For sure, let me know if you need help.

[NaganathanM]

Hello! Sorry I took long time. So, I've implemented a function that takes boxes - a list of coordinates and return the adjusted coordinates. Here, if length on x-axis is longer, then length on y-axis is adjusted, keeping the ymin the same. Similarly, xmin is kept the same and x-axis length is elongated if y-axis length is longer. Go through the code and give me your feedback:

def square_bounding_box(boxes):
    """
    The square_bounding_box function takes a list of x and y coordniates as arguments from the 'boxes' liat; calculates the longer side,
    and adjust the shorter side such that both lengths are equal to form a square, and return the coordniates
    boxes (list): A list of bounding box coordinates in the format [xmin, ymin, xmax, ymax]
    """
    if abs(boxes[1])-abs(boxes[0]) > abs(boxes[3])-abs(boxes[2]):
        boxes[3] = boxes[2] + abs(boxes[1])-abs(boxes[0])
    if abs(boxes[3])-abs(boxes[2]) > abs(boxes[1])-abs(boxes[0]):  
        boxes[1] = boxes[0] +  abs(boxes[3])-abs(boxes[2])         
        
    return boxes

#help(square_bounding_box)
if __name__ == "__main__":
    print(square_bounding_box([3,8,4,7]))
    print(square_bounding_box([6,9,2,9]))

[bw4sz]

good start, cleaning up code readability

def square_bounding_box(boxes: list) -> list:
    """
    Adjusts the bounding box coordinates to form a square.

    Args:
        boxes (list): A list of bounding box coordinates in the format [xmin, ymin, xmax, ymax]

    Returns:
        list: Adjusted bounding box coordinates to form a square
    """
    xmin, ymin, xmax, ymax = boxes

    width = xmax - xmin
    height = ymax - ymin

    if width > height:
        # Adjust height to match width
        ymax = ymin + width
    elif height > width:
        # Adjust width to match height
        xmax = xmin + height

    return [xmin, ymin, xmax, ymax]

if __name__ == "__main__":
    print(square_bounding_box([3, 8, 4, 7]))  # Example 1
    print(square_bounding_box([6, 9, 2, 9]))  # Example 2

[Karanveer266]

Hi! according to me this approach expands in only one direction (right or downward), shifting the object's center. This unidirectional expansion shifts the object's center position, which could potentially impact detection accuracy.

@ethanwhite @bw4sz If this issue hasn't been addressed yet, I'd like to volunteer to develop a center-preserving approach that expands boxes uniformly in all directions.

[Karanveer266]

The following is my solution for this issue:

def expand_bbox_to_square(bbox, image_width, image_height):
    """
    Expand a bounding box to a square by extending the shorter side.
    
    Parameters:
    -----------
    bbox : list or tuple
        Bounding box in format [x_min, y_min, width, height]
    image_width : int
        Width of the original image
    image_height : int
        Height of the original image
        
    Returns:
    --------
    list
        Square bounding box in format [x_min, y_min, width, height]
    """
    x_min, y_min, width, height = bbox
    
    center_x = x_min + width / 2
    center_y = y_min + height / 2
    
    side_length = max(width, height)
    
    new_x_min = center_x - side_length / 2
    new_y_min = center_y - side_length / 2
    
    new_x_min = max(0, min(new_x_min, image_width - side_length))
    new_y_min = max(0, min(new_y_min, image_height - side_length))
    
    if side_length > image_width:
        side_length = image_width
        new_x_min = 0
    
    if side_length > image_height:
        side_length = image_height
        new_y_min = 0
    
    return [new_x_min, new_y_min, side_length, side_length]

This implementation:

• Centers the square box on the original object by calculating the center point of the original bounding box directly. This preserves the object's position while expanding to a square.

• Handles boundaries more efficiently using min/max operations to constrain the coordinates in one step, rather than separate conditional branches.

• Simplifies the logic flow by first calculating the ideal position, then adjusting for boundaries, making the code more readable and maintainable.

• Handles extreme cases gracefully when the desired square is larger than the image dimensions.

[bw4sz]

Agreed, this looks like a reasonable PR to be added to https://github.com/weecology/DeepForest/blob/2e0b699bb0e1d0cb418d6b5d8504675559e85496/src/deepforest/model.py#L192, it should be an argument to allow users to turn off

write_crops(..., square_boxes=True)

1. Update the code
2. Update the docs
3. Show visual examples.
4. Submit PR. Thanks!

[Karanveer266]

Thanks for your response, I have already submitted PR #994 for this issue. Please checkout the PR and let me know if any more changes should me made.