Separate line and zone counters for separate classes

[zburq]

Search before asking

• [X] I have searched the Supervision issues and found no similar feature requests.

Question

My YOLO V8 is detecting & classifying several classes, say classes = [2, 3, 5, 7]:

I then follow examples by @SkalskiP and do something like:

line_counter = LineZone()
results = model.track(source=source_video, classes=classes, ... )
for result in results:
    detections = Detections.from_yolov8(result)
    detections.tracker_id = result.boxes.id.cpu().numpy().astype(int)
    line_counter.trigger(detections=detections)

Finally I can return and use line_counter.in_count and line_counter.out_count for other tasks.

Currently, LineZone().trigger will +=1 the in/out counts if a detection of any class crosses the line.

I'd like to keep a running tally of in/out counts for each class. Two possibilities:

• instantiate multiple LineZone objects, one for each class, or
• a single LineZone object to keep track of which class triggered how many counts.

I have modified my local code to accomplish this for now, but this would be a useful feature.

Perhaps LineZone.in_count and LineZone.out_count could old a vector corresponding to the class

Potential problem: Sometimes the classification of the tracked object flips between different classes during the tracking or line-crossing. My instinct is to use the majority class. For this, the line would have to be at the edge of the frame, so that we have seen all the classifications.

Anyone else solving a similar problem? Any better ideas?

Additional

No response

[github-actions[bot]]

Hello there, thank you for opening an Issue ! 🙏🏻 The team was notified and they will get back to you asap.

[SkalskiP]

Hi @zburq 👋🏻!

Sometimes, the classification of the tracked object flips between different classes during the tracking or line-crossing. My instinct is to use the majority class.

Yup, using 'majority class' was what I did in the past too. If your model is flickering between classes, that should be able to give you a better class assignment. Other things you could do:

• Train a model that would be more stable and robust.
• Moving window. Take the 'majority class from the last N frames.

Warning For this, the line would have to be at the edge of the frame so that we have seen all the classifications.

Unfortunately, this is a bad idea. For us to be able to say that object crossed the line, we need to be able to detect it on both sides of the line reliably. For example, it was on the left, and now it is on the right. If the line is too close to the edge of the frame in many cases, you won't get that final detection when the whole box makes it to the other side.

instantiate multiple LineZone objects, one for each class

This one feels better to me as you want need to write a lot of custom logic. Simply divide detections into groups and run each part through the associated line counter.

detections = Detections.from_yolov8(result)
detections_0 = detections[detections.class_id == 0]
line_counter_0.trigger(detections=detections_0)

[maddust]

@zburq i have accomplished determining what object and class crossed the line by checking the changes in the tracker state .for each line_counter

once the state changes from True to False or viceversa depending on the direction the object crossed you can then extract the class of that object based on the tracked_id. i will share my implementation to see if it helps.

[SkalskiP]

@maddust yup, that is a good solution. Would it be helpful if LineZone.trigger worked like PolygonZone.trigger and returned bool numpy.array? That would show you which Detections entry crossed the line.

[maddust]

@SkalskiP that would be great. I also have noticed some weird behavior when you have multiple lines . Sometimes the object gets counted even if it is outside of the line. A good test to replicate this is having two line counters collinear to each other .

[SkalskiP]

In that case, I'll keep this issue open. And add an enhancement label.

[SkalskiP]

@maddust what version of supervision do you run?

[maddust]

@maddust what version of supervision do you run? @SkalskiP im running the latest release 0.6 here an unedited example video , please fast forward it i did not edit it https://youtu.be/ZCQ8lITxqGg

[SkalskiP]

ooooh you have no idea how proud I am to see you use supervision to build an analytics system like that. 💜

Second of all, I understand what is happening. You are genuinely battle-testing that feature. Take a look below 👇. It looks like, and according to our logic, lines are endless.

We would need to add a second condition to our logic that, at the moment when the object crosses the line, it is between two perpendicular (invisible) lines. And only count crossing if detection is in the right positioning.

[SkalskiP]

@maddust I want to work on the fix for the bug. It would be awesome if you could share your line-counting logic and a short video example. Would that be possible?

[maddust]

@maddust I want to work on the fix for the bug. It would be awesome if you could share your line-counting logic and a short video example. Would that be possible?

hey @SkalskiP please find in the following repo my code for the line counters , basically im using supervision logic the only thing different is that im using a config.json file to set each line parameter and performing a for loop in the code so i dont have to rewrite code for each configuration .

https://github.com/maddust/Supervision-Line-Counter

btw thank you and the roboflow team for sharing this great project !, glad to help at least testing things out ! .

for the video please let me know what you would like me to show apart from the one i shared.

[SkalskiP]

You shared processing result. I’d love to have raw video that I can use to reproduce your experiment.

[maddust]

You shared processing result. I’d love to have raw video that I can use to reproduce your experiment. @SkalskiP ah Sure ! I've added a vide_download.txt in the repo please open and get the link from there.

[SkalskiP]

Thanks a lot 👍🏻 I'll try to work on some fix. I hope I'll be able to work on it soon. But it can take me few days :/

[maddust]

@SkalskiP i was able to fix the issue by adding this logic to the LineZone Tigger . please review . it would be nice to also be able to select which anchor Position will trigger the count for now i choose the bottom center .since i was no able to figure out how to make a parameter as how it is in PolygonZone. hope it helps. :) `class LineZone: """ Count the number of objects that cross a line. """

def __init__(self, start: Point, end: Point):
    """
    Initialize a LineCounter object.

    Attributes:
        start (Point): The starting point of the line.
        end (Point): The ending point of the line.

    """
    self.vector = Vector(start=start, end=end)
    self.tracker_state: Dict[str, bool] = {}
    self.in_count: int = 0
    self.out_count: int = 0
def is_within_line_segment(self, point: Point, margin: float = 2) -> bool:
    """
    Check if a point is within the line segment.

    Attributes:
        point (Point): The point to be checked.
        margin (float): Additional margin added to the line segment boundaries.

    Returns:
        bool: True if the point is within the line segment, False otherwise.
    """
    x_within = min(self.vector.start.x, self.vector.end.x) - margin <= point.x <= max(self.vector.start.x, self.vector.end.x) + margin
    y_within = min(self.vector.start.y, self.vector.end.y) - margin <= point.y <= max(self.vector.start.y, self.vector.end.y) + margin

    return x_within and y_within


def trigger(self, detections: Detections):
    """
    Update the in_count and out_count for the detections that cross the line.

    Attributes:
        detections (Detections): The detections for which to update the counts.

    """
    for xyxy, _, confidence, class_id, tracker_id in detections:
        # handle detections with no tracker_id
        if tracker_id is None:
            continue

        # we check if the bottom center anchor of bbox is on the same side of vector
        x1, y1, x2, y2 = xyxy
        anchor = Point(x=(x1+x2)/2, y=y2)  # Bottom center point of bounding box

        # Check if anchor is within the line segment
        if not self.is_within_line_segment(anchor):
            continue

        tracker_state = self.vector.is_in(point=anchor)

        # handle new detection
        if tracker_id not in self.tracker_state:
            self.tracker_state[tracker_id] = tracker_state
            continue

        # handle detection on the same side of the line
        if self.tracker_state.get(tracker_id) == tracker_state:
            continue

        self.tracker_state[tracker_id] = tracker_state
        if tracker_state:
            self.out_count += 1
        else:
            self.in_count += 1`

[sceddd]

Hi @zburq, This is an issue for this problem; you can try this one. I'm overriding both Suppervision Lib and also with @maddust's help. This could be what you're looking for.

from supervision.geometry.core import Vector
from typing import Dict, Optional 
from typing import Dict

import cv2
import numpy as np

from supervision.detection.core import Detections
from supervision.draw.color import Color
from supervision.geometry.core import Point, Rect, Vector
from typing import Optional

class LineZoneFixed:
  def __init__(self, start: Point, end: Point,class_id: np.ndarray, grace_period: int = 2):
    self.vector = Vector(start=start, end=end)
    self.tracker_state: Dict[str, bool] = {}
    self.already_counted: Dict[str, bool] = {} # Flag for each tracker_id
    self.frame_last_seen: Dict[str, int] = {} # Last frame each tracker_id was encountered
    self.current_frame: int = 0 # Current frame counter
    self.in_class: Dict[str,int] = {x:0 for x in class_id}
    self.out_class: Dict[str,int] = {x:0 for x in class_id}
    self.in_count: int = 0
    self.out_count: int = 0
    self.grace_period = grace_period # Number of frames to wait before resetting
  def is_within_line_segment(self, point: Point, margin: float = 5) -> bool:
    """
    Check if a point is within the line segment.

        Attributes:
            point (Point): The point to be checked.
            margin (float): Additional margin added to the line segment boundaries.

        Returns:
            bool: True if the point is within the line segment, False otherwise.
    """
    x_within = min(self.vector.start.x, self.vector.end.x) - margin <= point.x <= max(self.vector.start.x, self.vector.end.x) + margin
    y_within = min(self.vector.start.y, self.vector.end.y) - margin <= point.y <= max(self.vector.start.y, self.vector.end.y) + margin

    return x_within and y_within
  def trigger(self, detections: Detections) -> np.ndarray:
    self.current_frame += 1  # Increment frame counter
    for xyxy, _, confidence, class_id, tracker_id in detections:
      # handle detections with no tracker_id
      if tracker_id is None:
        continue

      # we check if the bottom center anchor of bbox is on the same side of vector
      x1, y1, x2, y2 = xyxy
      anchor = Point(x=(x1+x2)/2, y=y2)  # Bottom center point of bounding box

      # Check if anchor is within the line segment
      if not self.is_within_line_segment(anchor):
        continue

      tracker_state = self.vector.is_in(point=anchor)

      # handle new detection
      if tracker_id not in self.tracker_state:
        self.tracker_state[tracker_id] = tracker_state
        self.already_counted[tracker_id] = False  # When the object appears, it has not been counted yet
        self.frame_last_seen[tracker_id] = self.current_frame  # Update last seen frame
        continue

      # handle detection on the same side of the line
      if self.tracker_state.get(tracker_id) == tracker_state:
        continue

      # check if this crossing has already been counted
      if self.already_counted.get(tracker_id, False):  # If it has been counted, we skip this
          # Check if grace period has passed since last encounter
        if self.current_frame - self.frame_last_seen.get(tracker_id, 0) > self.grace_period:
            self.already_counted[tracker_id] = False  # Reset after grace period
        else:
            continue

      self.tracker_state[tracker_id] = tracker_state
      self.already_counted[tracker_id] = True  # After counting, we mark this crossing as already counted
      self.frame_last_seen[tracker_id] = self.current_frame  # Update last seen frame
      if tracker_state:
        self.in_class[class_id] += 1
      else:
        self.out_class[class_id] +=1
    print(self.in_class,"\t",self.out_class)
    self.in_count = sum(self.in_class.values())
    self.out_count = sum(self.out_class.values())

[epochDVKHN]

@SkalskiP I cannot find anything in the documentation talking about LineZone (I only see PolygonZone). Can you please add it?

[SkalskiP]

Hi, @ElNoSabe322 👋🏻! We want to redesign LineZone in one of our upcoming releases. And for that reason, it is left undocumented. Anything in particular that you would like to ask?

[epochDVKHN]

Hi, @ElNoSabe322 👋🏻! We want to redesign LineZone in one of our upcoming releases. And for that reason, it is left undocumented. Anything in particular that you would like to ask?

Oh I see... yeah fair enough. Thank you for your reply @SkalskiP. I will await the documentation and the updated release (I do not have anything else to ask for now). 👍

[falkaabi]

I think LineZone should have separate functionality for triggers and counting. It's currently doing too much. A linezone should only be concerned with checking if a tracked object has crossed the line or not. Thus having the ability to customize so many things in downstream tasks. For example, have the trigger call another function which could for example increase a counter, or return an instance of the object that crossed the line to do further processing. We should also have the ability to choose which anchor points should trigger a line crossing:

1. All 4 corner points have crossed the linezone (which is the current implementation)
2. Center point of bounding box has crossed the linezone
3. Center point of one of the bounding box sides has crossed the line (north, south, east, or west)
4. Any single corner point has crossed the linezone (instant trigger once bounding box touches the line)

For example, if one would like to track vehicles crossing a line, then do ANPR when the linezone is triggered to get accurate license plate recognition, it would be much easier if the above functionalities were separate and customizable.

[SkalskiP]

@maddust this problem was just solved

that would be great. I also have noticed some weird behavior when you have multiple lines . Sometimes the object gets counted even if it is outside of the line. A good test to replicate this is having two line counters collinear to each other .

https://github.com/roboflow/supervision/pull/735

[SkalskiP]

This issue has been split into two separate ones - https://github.com/roboflow/supervision/issues/790 and https://github.com/roboflow/supervision/issues/791.