How to access camera intrinsics parameters

[nikhilsawlani]

Hey everyone, I'm trying to develop an AR CV application on top of WebXR, it needs camera intrinsic parameters such as focalLength and principalPoint of camera.

is there any API or method through which I can access these parameters for each device running on the web?

[klausw]

If I'm understanding it right, you should be able to calculate camera intrinsics from the projection matrix and viewport provided alongside the viewer pose. I couldn't find a specific example in a quick search, but it seems that the camera matrix used in CV is closely related to the GL projection matrix, except for using NDC coordinates (-1..1 range) instead of pixel values. Is that correct?

FYI, the WebXR AR API doesn't currently provide access to raw camera images, that's currently being discussed as an extension feature, see for example https://github.com/bialpio/webxr-raw-camera-access/blob/master/explainer.md

On Fri, Nov 27, 2020, 00:42 Nikhil [email protected] wrote:

Hey everyone, I'm trying to develop an AR CV application on top of WebXR, it needs camera intrinsic parameters such as focalLength and principalPoint of camera.

is there any API or method through which I can access these parameters for each device running on the web?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/immersive-web/webxr/issues/1147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKT5X6E5I7CGL2YLKCZY5LSR5Q5TANCNFSM4UET4CJQ .

[nikhilsawlani]

Hi, @klausw Thanks for the quick response,

I will look for the conversion of the GL projection matrix to camera intrinsic matrix, I found an example here which was able to access camera frame data and then render on the cube, so I thought its possible now to get the camera image, isn't it?

[klausw]

That example is based on the experimental feature I mentioned, see https://www.chromestatus.com/feature/5759984304390144 for more information about the status in Chrome It requires the "WebXR Incubations" flag in chrome://flags and the API may change based on feedback. It would be very useful to know if it meets requirements for CV integration, please let us know about your results.

On Fri, Nov 27, 2020, 23:45 Nikhil [email protected] wrote:

Hi, @klausw https://github.com/klausw Thanks for the quick response,

I will look for the conversion of the GL projection matrix to camera intrinsic matrix, I found an example here https://storage.googleapis.com/chromium-webxr-test/r827010/proposals/camera-access-barebones.html which was able to access camera frame data and then render on the cube, so I thought its possible now to get the camera image, isn't it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/immersive-web/webxr/issues/1147#issuecomment-735052065, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKT5X2VL5XKBKQVMSCYDX3SSCTALANCNFSM4UET4CJQ .

[nikhilsawlani]

Camera intrinsic are mostly represented in image pixels, I found an example which shows how to use intrinsic camera matrix in OpenGL http://ksimek.github.io/2013/06/03/calibrated_cameras_in_opengl/

mainly it is: Proj=NDC×Persp where the intrinsic camera matrix describes a perspective projection

[esahin90]

hello everybody, I was trying to achieve something similar, where I want to extract the perspective projection matrix for using it within ROS, which uses senser_msgs/CameraInfo. This expects the matrix to be in this format:

# Projection/camera matrix
#     [fx'  0  cx' Tx]
# P = [ 0  fy' cy' Ty]
#     [ 0   0   1   0]

where Tx and Ty should be zero for monocular cameras, when using the depth estimation from the depth sensing module.

Would be nice to find a solution to extract the information without OpenGL. I would appreciate any help :)

[klausw]

FYI, @bialpio and me have added a section about obtaining camera extrinsics and intrinsics to the WebXR Raw Camera Access explainer: https://github.com/immersive-web/raw-camera-access/blob/main/explainer.md#obtaining-camera-extrinsics-and-intrinsics

Sample JS code for this is included in the test page, see the source code for getCameraIntrinsics.

Example output for a Pixel 2 XL in landscape mode:

projection: [1.5731755495071411, 0, 0, 0, 
             0, 3.146350622177124, 0, 0,
             0, 0, -1.0001999139785767, -1, 
             0, 0, -0.2000199854373932, 0]
intrinsics: u0=960 v0=480 ax=1510.2485275268555 ay=1510.2482986450195 gamma=0 for viewport {width=1920,height=960,x=0,y=0}

We'd appreciate feedback if this meets your needs and works as expected. I think the printed values look plausible, but I don't know if anyone has published an example where this is used with a CV library or similar.