How to convert LLFF camera poses to Pytorch3D cameras?

[w2kun]

Hi, thanks for your great work! I recently tried to train a NeRF model using LLFF data, but encountered some problems when convert the LLFF camera poses to Pytorch3D cameras. My code is like that:

# load_llff_data is from https://github.com/bmild/nerf/blob/18b8aebda6700ed659cb27a0c348b737a5f6ab60/load_llff.py#L243
images, poses, _, _, _ = load_llff_data(basedir, factor=8, recenter=True, bd_factor=.75, spherify=False)
focal = poses[:, -1, -1]  # (n,)
rt = poses[:, :, : -1].copy()  # (n, 3, 4)
rt[:, :, [0, 2]] *= -1.0  # flip x and z
rotation = rt[:, :, : -1]  # (n, 3, 3), cam2world->world2cam and T@P->P@T
translation = rt[:, :, -1] * -1.0  # (n, 3), times -1.0 to perform cam2world->world2cam
focal_ndc = focal * 2.0 / min(images.shape[1: 3])
cameras = PerspectiveCameras(
    focal_length=focal_ndc,
    R=rotation,
    T=translation,
    in_ndc=true
)  # the code of numpy->tensor is omitted

Meanwhile, I modify the code in _xy_to_ray_bundle to make the direction point to -z.

The above code doesn't work well, so I hope you can help me to correct it. Thanks for your kind help!

[MatrixZzu]

Hi, thanks for your great work! I recently tried to train a NeRF model using LLFF data, but encountered some problems when convert the LLFF camera poses to Pytorch3D cameras. My code is like that:

# load_llff_data is from https://github.com/bmild/nerf/blob/18b8aebda6700ed659cb27a0c348b737a5f6ab60/load_llff.py#L243
images, poses, _, _, _ = load_llff_data(basedir, factor=8, recenter=True, bd_factor=.75, spherify=False)
focal = poses[:, -1, -1]  # (n,)
rt = poses[:, :, : -1].copy()  # (n, 3, 4)
rt[:, :, [0, 2]] *= -1.0  # flip x and z
rotation = rt[:, :, : -1]  # (n, 3, 3), cam2world->world2cam and T@P->P@T
translation = rt[:, :, -1] * -1.0  # (n, 3), times -1.0 to perform cam2world->world2cam
focal_ndc = focal * 2.0 / min(images.shape[1: 3])
cameras = PerspectiveCameras(
    focal_length=focal_ndc,
    R=rotation,
    T=translation,
    in_ndc=true
)  # the code of numpy->tensor is omitted

Meanwhile, I modify the code in _xy_to_ray_bundle to make the direction point to -z.

The above code doesn't work well, so I hope you can help me to correct it. Thanks for your kind help!

Hi, did you convert successfully?

[w2kun]

not yet.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[sergeyprokudin]

hi folks, any update on this by chance?

[kjchalup]

Take a look at https://github.com/facebookresearch/pytorch3d/blob/main/pytorch3d/implicitron/dataset/single_sequence_dataset.py#L163 as well as how that is called in llff_dataset_map_provider.py, maybe that will help?

[sergeyprokudin]

Hi @kjchalup, thanks a lot, that worked! Folllowing a suggestion from Nikhila here, I've also created a small Colab script which is doing the conversion of the NeRF LLFF Fern data and comparing it to the Pytorch3d data version already provided by Meta. Maybe it will help other researchers investigating the same issue.

Thanks again, ~Sergey