Loading instant-ngp data with missing fields

[cpacker]

Describe the bug

I tried running ns-train nerfacto on an "instant-ngp-ready" data directory, but get the following error:

  File "anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/dataparsers/nerfstudio_dataparser.py", line 151, in _generate_dataparser_outputs
    fx=float(meta["fl_x"]),
KeyError: 'fl_x'

Seems like this is because instant-ngp does some conversions (where possible) to populate missing fields in the input transforms.json file, see: https://github.com/NVlabs/instant-ngp/blob/ce8417509f3d642efe93f230ed68f804de0f8f31/src/nerf_loader.cu#L271-L282

Ideally, nerfstudio can do the same sort of conversions so that any transforms.json file that loads into instant-ngp can also load into nerfstudio.

To Reproduce

My transforms.json file has the following fields, and loads fine into my build of instant-ngp (happy to provide the full folder if necessary):

{
    "camera_angle_x": 0.7853981633974483,
    "aabb_scale": 16,
    "scale": 0.15,
    "offset": [
        0.5,
        1.3,
        0.2
    ],
    "frames": [
   ]

[cpacker]

Thanks for patching @akristoffersen!

Not sure if this is WIP, but there are a few more missing field conversions that happen that in instant-ngp that don't happen in the latest code on main. AFAIK, the only required fields in an instant-ngp json file are camera_angle_x (or some other info to infer FOV - w/o camera_angle_x you get the following error: ERROR Uncaught exception: Couldn't read fov.), and frames.

Currently I get a runtime access error on k1, but I I think all of the following fields need to be inferred upon omission to match instant-ngp json requirements:

• meta["k1"], meta["k2"]
  • [nerfstudio] accessed directly here: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/data/dataparsers/instant_ngp_dataparser.py#L85
  • [instant-ngp] accessed with fallback here: https://github.com/NVlabs/instant-ngp/blob/ce8417509f3d642efe93f230ed68f804de0f8f31/src/nerf_loader.cu#L200-L212
• meta["p1"], meta["p2"]
  • [nerfstudio] accessed directly here: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/data/dataparsers/instant_ngp_dataparser.py#L85
  • [instant-ngp] accessed with fallback here: https://github.com/NVlabs/instant-ngp/blob/ce8417509f3d642efe93f230ed68f804de0f8f31/src/nerf_loader.cu#L214-L226
• meta["cx"], meta["cy"]
  • [nerfstudio] accessed directly here: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/data/dataparsers/instant_ngp_dataparser.py#L102-L103
  • [instant-ngp] accessed with fallback here: https://github.com/NVlabs/instant-ngp/blob/ce8417509f3d642efe93f230ed68f804de0f8f31/src/nerf_loader.cu#L228-L234
• meta["h"], meta["w"]
  • [nerfstudio] accessed directly here: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/data/dataparsers/instant_ngp_dataparser.py#L105-L106
  • [instant-ngp] accessed with fallback here: https://github.com/NVlabs/instant-ngp/blob/ce8417509f3d642efe93f230ed68f804de0f8f31/src/nerf_loader.cu#L495-L497
• meta["aabb_scale"]

[cpacker]

Reopening based on the above comment, feel free to close if this isn't a planned feature.

[akristoffersen]

Thank you for opening this back up! I'll get started on another patch ASAP