Add 2x Inference Slicer blocks

[LinasKo]

Description

This PR adds the RoboflowDetectionSlicerBlock and RoboflowSegmentationsSlicerBlock blocks and respective unit tests, but not integration tests. If needed, I can add those after the lighthouse.

They can be found in the new folder, under inference/core/workflows/core_steps/supervision_tools/.

This is a draft PR as I want to address a few items mentioned in the Review suggestions section below.

Type of change

• [x] New feature (non-breaking change which adds functionality)

How has this change been tested, please provide a testcase or example of how you tested the change?

1. I ran the unit tests included in this PR. (no errors)
2. I generated the block pages and built the docs with mkdocs serve, observing that blocks appear in the blocks list.
3. I ran two python scripts simulating block usage and verified that returned detections are correct. Code provided below.
4. ⚠️ I did NOT run this inside an actual workflow.

simulate_detections_slicer_block.py

import os
import cv2
import numpy as np
import supervision as sv
from inference.core.managers.base import ModelManager
from inference.core.registries.roboflow import (
    RoboflowModelRegistry,
)
from inference.core.workflows.core_steps.supervision_tools.detections_slicer import RoboflowDetectionSlicerBlock
from inference.core.workflows.entities.base import Batch, ImageParentMetadata, WorkflowImageData
from inference.models.utils import ROBOFLOW_MODEL_TYPES


API_KEY = os.getenv("ROBOFLOW_API_KEY")
assert API_KEY is not None, "Please set the ROBOFLOW_API_KEY environment variable"


model_registry = RoboflowModelRegistry(ROBOFLOW_MODEL_TYPES)
model_manager = ModelManager(model_registry=model_registry)

block = RoboflowDetectionSlicerBlock(
    model_manager=model_manager,
    api_key=API_KEY
)

img_path = "cat.png"
image = cv2.imread(img_path)
# image = np.zeros((192, 168, 3), dtype=np.uint8)


images = Batch([
    WorkflowImageData(
        parent_metadata=ImageParentMetadata(parent_id="$inputs.image"),
        numpy_image=image,
    )
])

async def main():
    model_id = "yolov8n-640"
    box_annotator = sv.BoundingBoxAnnotator()
    label_annotator = sv.LabelAnnotator()

    res = await block.run_locally(
        images=images,
        model_id=model_id,
        class_agnostic_nms=None,
        class_filter=None,
        confidence=0.4,
        iou_threshold=0.4,
        slice_width=640,
        slice_height=640,
        overlap_ratio_width=0.2,
        overlap_ratio_height=0.2
    )

    detections = res[0]["predictions"]
    ann = image.copy()
    ann = box_annotator.annotate(ann, detections)
    ann = label_annotator.annotate(ann, detections)
    cv2.imwrite("det_local.png", ann)
    print(f"#####################\nLocal run result:")
    print(res)

    res = await block.run_remotely(
        images=images,
        model_id=model_id,
        class_agnostic_nms=None,
        class_filter=None,
        confidence=0.4,
        iou_threshold=0.4,
        slice_width=640,
        slice_height=640,
        overlap_ratio_width=0.2,
        overlap_ratio_height=0.2
    )

    detections = res[0]["predictions"]
    ann = image.copy()
    ann = box_annotator.annotate(ann, detections)
    ann = label_annotator.annotate(ann, detections)
    cv2.imwrite("det_remote.png", ann)
    print(f"#####################\nRemote run result:")
    print(res)

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

simulate_segmentations_slicer_block.py

import os
import cv2
import numpy as np
import supervision as sv
from inference.core.managers.base import ModelManager
from inference.core.registries.roboflow import (
    RoboflowModelRegistry,
)
from inference.core.workflows.core_steps.supervision_tools.segmentations_slicer import RoboflowSegmentationSlicerBlock
from inference.core.workflows.entities.base import Batch, ImageParentMetadata, WorkflowImageData
from inference.models.utils import ROBOFLOW_MODEL_TYPES


model_registry = RoboflowModelRegistry(ROBOFLOW_MODEL_TYPES)
model_manager = ModelManager(model_registry=model_registry)

block = RoboflowSegmentationSlicerBlock(
    model_manager=model_manager,
    api_key=os.getenv("ROBOFLOW_API_KEY")
)

img_path = "cat.png"
image = cv2.imread(img_path)


images = Batch([
    WorkflowImageData(
        parent_metadata=ImageParentMetadata(parent_id="$inputs.image"),
        numpy_image=image,
    )
])

async def main():
    model_id = "yolov8n-seg-640"
    mask_annotator = sv.MaskAnnotator()
    label_annotator = sv.LabelAnnotator()

    res = await block.run_locally(
        images=images,
        model_id=model_id,
        class_agnostic_nms=None,
        class_filter=None,
        confidence=0.2,
        mask_decode_mode="accurate",
        tradeoff_factor=0.0,
        iou_threshold=0.2,
        slice_width=640,
        slice_height=640,
        overlap_ratio_width=0.2,
        overlap_ratio_height=0.2
    )

    detections = res[0]["predictions"]
    ann = image.copy()
    ann = mask_annotator.annotate(ann, detections)
    ann = label_annotator.annotate(ann, detections)
    cv2.imwrite("seg_local.png", ann)
    print(f"#####################\nLocal run result:")
    print(res)

    res = await block.run_remotely(
        images=images,
        model_id=model_id,
        class_agnostic_nms=None,
        class_filter=None,
        confidence=0.2,
        mask_decode_mode="accurate",
        tradeoff_factor=0.0,
        iou_threshold=0.2,
        slice_width=640,
        slice_height=640,
        overlap_ratio_width=0.2,
        overlap_ratio_height=0.2
    )

    detections = res[0]["predictions"]
    ann = image.copy()
    ann = mask_annotator.annotate(ann, detections)
    ann = label_annotator.annotate(ann, detections)
    cv2.imwrite("seg_remote.png", ann)

    print(f"#####################\nRemote run result:")
    print(res)


if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Review suggestions

Here's a few areas that need to be looked at:

1. Naming:

• RoboflowSegmentationsInferenceSlicer, RoboflowDetectionsInferenceSlicer are a mouthful.
• I don't think it should reside in supervision_tools.
• I matched the plurality of names to Detections, but SegmentationSlicer would sound better for me.
• file names segmentations_slicer.py and detections_slicer.py.

2. In segmentations_slicer.py, when "points" are set and sv.mask_to_polygons is called, I've selected the first polygon. I need to think whether that's always the case.
3. class agnostic NMS is performed on detections for each slice, but I believe we should only rely on the slicer doing it.
4. supervision 0.21.0 adds both segmentation slicer support and non-max-merging. Non-max-merging is selection was not added here.
5. Shall I add some integration tests?

Any specific deployment considerations

1. Run the pre-commit linter.
2. Test inside a workflow (JSON or UI) before integration.

Docs

• [x] Docs updated? What were the changes:

Two new blocks are present in the Blocks list. CTRL+F for slicer.

[LinasKo]

I'm bringing the SlicerBlock code I had up-to-speed.

@grzegorz-roboflow, with respect to your comments:

• We can't change the params to InferenceSlicer as it is in supervision, unfortunately. We can, however, wrap the implementation with custom logic, if that makes it more intuitive for the users. Do you think we should do that?
• I've removed the back-and-forth conversion. In this case, when sv.Detections are obtained, they'll be kept.
• I've added an integration test for the slicer, using an image from assets.

@grzegorz-roboflow or @PawelPeczek-Roboflow, would you have time to take a look at this?

[LinasKo]

I need some help. I don't understand why in my integration test I am getting images as a single WorkflowImageData rather than Batch[WorkflowImageData].

[LinasKo]

As discussed with @PawelPeczek-Roboflow, I'm handing this off to him.

[PawelPeczek-Roboflow]

closing in favour of https://github.com/roboflow/inference/pull/574

[LinasKo]

Cool cool. I was just looking at it, thinking whether you got it working