Blurry image with long focal length cam

[pooncs]

Hi great work on this repo,

Previously I was using Instant-ngp to train aerial images (98 images). It worked well in instant-ngp (psnr: 25 after 30k iterations), however, when I was training the same set of image in nerfstudio, the results never got more than psnr: 18. I've tried other methods too (nerfacto, tensorf,..), they all looked blurry. with nothing more than 18.

To attempt debugging this issue, I have tried running colmap from nerfstudio, the result was still blurry, then I took the same set of data and ran it on nerfacc, which the result was close to instant-ngp (psnr: ~24). So I was wondering if it is the long focal length or something being clipped or something else?

Thanks.

Here is a sample of the transform.json that was produced by the nerfstudio's colmap

"fl_x": 7516.557593687107, "fl_y": 7537.338324771301, "cx": 511.56096564540155, "cy": 268.4251482048674, "w": 1024, "h": 540, "camera_model": "OPENCV", "k1": -0.014968006153574012, "k2": -0.0005643351843003853, "p1": -0.0009081580152328081, "p2": 0.0001238689035023915, "frames": [ { "file_path": "images/frame_00098.png", "transform_matrix": [ [ 0.6162682283213603, 0.7706546234163679, 0.16218792238828256, 0.5686785746987137 ], [ -0.5231373264832101, 0.24665259555975744, 0.8157756031799638, 4.511451631367819 ], [ 0.5886771682354011, -0.587583141785578, 0.5551623574124407, -0.3244527595299395 ], [ 0.0, 0.0, 0.0, 1.0 ] ] },

Here is the tensorboard output after training.

[tancik]

It's possible that the pose optimization is unstable when working with such long focal lengths. Maybe try disabling it to see if it helps, --pipeline.datamanager.camera-optimizer.mode off

[pooncs]

Thanks for the quick reply, I've tried with and without the camera optimizer, which it didn't seemed to work :/

[tancik]

Can you try increasing the "near" plane, --pipeline.model.near-plane it defaults to 0.05, you might need larger, you can also try increasing the far plane, --pipeline.model.far-plane.

Other parameters to try, --optimizers.proposal-networks.optimizer.lr 0.01 --pipeline.model.distortion-loss-mult 0

[pooncs]

didn't seemed to work either :/

[f-dy]

Ok so here are my explanations. First of all, calibrating long focal length with COLMAP may give you a good reprojection error, but the actual optical center of the camera is very unreliable. In fact, cameras with long focal length may be very close to orthographic cameras, which have their optical center at infinity. Then, there is the way nerfstudio computes scene scale, which is based on those camera optical centers by default: the [-1,1]^3 cube should contain the camera optical centers. If you also use the default ("poses") centering method, then the scene center will also be set to the centroid of these camera centers, which may be high above ground in your case. The "focus" scene centering method should set a center closer to the ground. But still, if your scene is, say, 300mx300m, but the cameras are 2000m over the ground, the scene scale will be wrongly computed and the nerf resolution will be bad. What I recommend is:

• use the "focus" scene centering method
• set the scene scale yourself, so that your actual scene is mainly inside [-1,1]^3. You'll have to fiddle with some tools for that. Maybe the nerfstudio viewer could be used. Maybe the COLMAP gui. Then you can set center-method=focus, auto_scale_poses=False and scene_scale=your_value in ns-train, e.g. ns-train ... --pipeline.datamanager.camera-optimizer.mode off nerfstudio-data --center-method focus --auto-scale-poses False --scene-scale 0.1.
• Once you've validated anything, and the model trains correctly, if you want to relaunch training using all images, add --train-split-fraction 1.0 (you won't be able to run ns-eval)

Now the nerfacto model should have more resolution on your scene.