efficient_pnp() function usage

[gsvirskiy]

Hello.

I have some difficulties in understanding how the efficient_pnp() function works and what it accepts as arguments. My setting is the following: I have 2d points in the image space, i.e. pixel coordinates, and corresponding 3d coordinates in the object coordinate frame. I want to find the camera orientation relative to that object using PnP.

In order to do this, I first convert the 2d pixel coordinates to the camera rays using a camera intrinsics matrix K: x2d_cam = K^-1 @ x2d

Then, I do the perspective projection and take the first two components of the resulting points: x2d_cam = x2d_cam[..., :2] / x2d_cam[..., 2] , to obtain "uncalibrated 2D points".

According to the documentation of the function, this is what this function is expected to have as inputs.

the efficient_pnp() function is then called with the following args: efficient_pnp(x3d, x2d_cam)

First observation, it doesn't work at all as cv.solvePnP function, the results are quite different. And the other thing is that I always got warnings that some points are behind the camera. i.e. have z < 0.

What am I doing wrong and need to change in my approach?

Thanks.