Contours sorting and mouse movement

[codeYazar]

First of all, your work is working very well. And I'm new on this. I'm trying to understand codes.

what does it mean , especially [-2] in moments = cv2.moments(contours[-2])

I also want to ask you how to match the coordinates on the webcam, on the PC screen. Thanks

[antoinelame]

Hi @codeYazar

what does it mean , especially [-2] in moments = cv2.moments(contours[-2])

The algorithm detects faces, eyes and then processes the frame to get that:

The black block is the iris. We use it to calculate the centroid (I consider that is the pupil position). First, it's necessary to detect this contour:

_, contours, _ = cv2.findContours(self.iris_frame, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

It returns a list of several contours:

• the contour of the entire frame
• the contour of the iris
• the contours of other small black blocks if the binarization was not optimal

Of course, we want to get only the iris contour. So to do that, the algorithm sorts the contours list by size:

contours = sorted(contours, key=cv2.contourArea)

The contours are now sorted from the smallest to the largest. So the last one is the contour of the entire frame and the second to last is the contour of iris.

So we can access iris contour like so: contours[-2]

I also want to ask you how to match the coordinates on the webcam, on the PC screen.

I don't understand well your question. Do you want to use it to move your mouse?

[codeYazar]

Yeah. I want to move the mouse with my eyes. But the head will not be fixed. He should let the head move.

[antoinelame]

To have the relative coordinates of the pupils, without the position of the head having any impact, you can change:

def pupil_left_coords(self):
        """Returns the coordinates of the left pupil"""
        if self.pupils_located:
            x = self.eye_left.origin[0] + self.eye_left.pupil.x
            y = self.eye_left.origin[1] + self.eye_left.pupil.y
            return (x, y)

to

def pupil_left_coords(self):
        """Returns the coordinates of the left pupil"""
        if self.pupils_located:
            x = self.eye_left.pupil.x
            y = self.eye_left.pupil.y
            return (x, y)

[wittynerd]

I tried using horizontal_ratio and vertical_ratio to move the mouse cursor (using pyautogui library) as in,

`` X,Y = gaze.horizontal_ratio(),gaze.vertical_ratio() w , h = pyautogui.size() #Returns the size of the screen

try:
    x = int(w*X)
    y = int(h*Y)
   
except TypeError:
    x , y = None,None
 
print(x,y)
pyautogui.moveTo(x,y)    #Moves the mouse cursor to (x,y)``

The performance isn't good. Is there any better way or a calibration technique I'm missing?

[antoinelame]

Hi @wittynerd, gaze.horizontal_ratio() and gaze.vertical_ratio() return ratios to know the gaze direction. It doesn't return any coordinates, and you shouldn't try to get coordinates from it. Instead, you should use gaze.pupil_left_coords() and gaze.pupil_right_coords(). Good luck!