Help using Dlib coordinates

[ccastelano]

First of all, congratulations for your good job! My question is: Is there any way to grab the coordinates (x, y) of the points detected by the algorithm, inside the "SessionHandler.swift" file? (more specifically inside the "captureOutput" function). I'm trying to figure out how to do this for a few days, but I couldn't get any further, because I'm having difficuties to understand Objective C and adapt the code. Thank you for sharing!

[zweigraf]

Yes you could return the calculated coordinates from the doWork method in DlibWrapper.

[ccastelano]

It worked perfectly, man!

However, I came across with one last "coordinate conversion" issue.

The code after the modification seems like:

-(CGPoint)doWorkOnSampleBuffer:(CMSampleBufferRef)sampleBuffer inRects:(NSArray<NSValue *> *)rects {

    CGPoint mypoint = CGPointMake(0, 0); // DECLARING MY POINT

    (...)

        for (unsigned long k = 0; k < shape.num_parts(); k++) {
            dlib::point p = shape.part(k);
            
            if (k == 36) {
                mypoint = CGPointMake(p.x(), p.y()); // MY POINT OF INTEREST
            }

    (...)

    return mypoint;
}

Now, inside my ViewController.swift, I declared an global variable for "myView" as follow:

var myView = UIView()

(...)

override func viewDidLoad() {
        super.viewDidLoad()
        
        myView = UIView(frame: CGRect(x: 0, y: 0, width: 20, height: 20))
        view.addSubview(myView)
        view.bringSubview(toFront: myView)
 }

And finally, inside my SessionHandler.swift file, I did like:

func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {

    (...)

            var myPoint = wrapper?.doWork(on: sampleBuffer, inRects: boundsArray)
            print("myPoint = ", myPoint)
            
            DispatchQueue.main.async(execute: {
                
                myView.center = CGPoint(x: ((myPoint?.x)!/2), y: ((myPoint?.y)!/2))
                myView.backgroundColor = .red
                
            })
}

Then, I noticed that the position of the points are not coinciding, and things seems mislocated.

I think I need to make some coordinate (x,y) conversion.

Any ideia?

Thanks!

[zweigraf]

If I understand the problem correctly, yes you need some coordinate conversion.

In this line: https://github.com/zweigraf/face-landmarking-ios/blob/master/DisplayLiveSamples/SessionHandler.swift#L72 we first convert the metadata object into some other format depending on camera orientation and stuff like that.

Then, in this line: https://github.com/zweigraf/face-landmarking-ios/blob/master/DisplayLiveSamples/DlibWrapper.mm#L87 we are converting those coordinates into Dlib suitable coordinates.

If you want to use the points again in SessionHandler, after calculating your point, you first need to reverse the Dlib conversion from DlibWrapper (instead of iOS-> Dlib make it Dlib->iOS), and then maybe reverse the camera conversion from SessionHandler.

[harshchitrla]

@zweigraf Any idea how do I convert Dlib to iOS points ? Anyone tried to figure out.

[harshchitrla]

@ccastelano hi, did you manage to solve "coordinate conversion" issue

[leynercastillo]

@harshchitrla hi. The coordinates returned by Dlib are the coordinates of the original image. You only need to convert the original image coordinate to you UIImageView or containerView coordinates. That depend of you content mode and the container size.

[jirkaskotak]

Hey, has anyone found a solution to this?