Pinhole 2D view has focal issue when projecting 3d with undistorted images, but world works

[OlivierLDff]

Describe the bug With some pinhole model, 3d projection on image plane is buggy. At first I thought it was a problem of my matrices, but no this is really a bug in the viewer. The "world view" gives correct results.

To Reproduce Here are my 2 rrd files:

• The buggy one: https://drive.google.com/file/d/1HF04o7kvnwB60SkwMPAinXIB5NULduYF/view?usp=sharing,
• The one working: https://drive.google.com/file/d/1N3Esx-GE9TE9scu5p-PR287N9hwQgsd8/view?usp=sharing

The data has been produced from public panoptic dataset. The first file uses an undistort alpha of 0.0 and produce buggy visualization. The second file uses an undistort alpha of 1.0 and produce good result.

I have no idea what is causing that, as I can only see this problem in rerun viewer.

Expected behavior The focal to be correct

Screenshots Undistort 0, as you can see all 3d keypoints and bboxes are "off":

Undistort 1:

As a proof this is a problem in the 2d view, when tracing a ray it's correctly going on the 3d model:

Rerun version 0.23

[Wumpf]

thanks for the repro case and excellent issue description!

[Wumpf]

had a quick first look and I believe the problem is our lack of handling for anamorphic cameras in the 2D view: the broken looking rrd has different focal length for x and y whereas the working one has x==y on focal length

Relevant code: https://github.com/rerun-io/rerun/blob/65bf078f22592b88a918456905382ac19eab7bf0/crates/viewer/re_view_spatial/src/ui_2d.rs#L338

[Wumpf]

bit confused why it works out just fine in the 3d view - when double clicking on the 3d camera and moving the image plane out (via overrides in the selection panel) such that it's behind the 3d objects things work out still fine. And we definitely don't handle anamorphic cameras there either. So maybe that's not actually relevant as all the necessary projection already happend in the data? Both rrds have fov x/y divergent, but it is noteworthy that one the seems-to-work one diverges very little. Either way, even if the issues was a missinterpretation of the data, the fact that the camera ray lines up differently with the 3d objects in 2d and 3d means there's a bug in the Rerun viewer.

Also worth pointing out that the dataset also comes in distorted (as posted in the original issue, this run is with undistorted data, just with different parameters)

• see http://domedb.perception.cs.cmu.edu/develop/tools.html
• making this related to: https://github.com/rerun-io/rerun/issues/2499

[OlivierLDff]

Yes I undistorted both videos and computed the new camera matrix (using opencv). I had to do that as like you pointed it out, distortions isn't supported yet by rerun.

[ndg-genweb]

I am facing a similar issue, where the projection works correctly for fx==fy but not otherwise

[martin-meinke-ai]

Same here :D just wasted hours looking at my intrinsics and doubting my sanity. everything works fine if fx==fy, but not if they are different.