Getting scaled exports

[Baptiste-AIST]

Describe the bug Hi everyone, I'm not too sure if it's a bug, but I wanted to know how the generated point cloud or meshes are scaled when exporting. I tried to compare the sizes from instant-ngp, nerfstudio and COLMAP and they differ between each other (using the same transforms.json). Is there a place when training or exporting where things get scaled?

Thank you very much!

Expected behavior At least I was expecting nerfstudio to generate a similar scaling as COLMAP or instant-ngp

Screenshots

Marker size from instant-ngp marshing cubes mesh export

0.278130m

Of course the measures are not so precise, but we can see the order of magnitude difference.

[tancik]

The dataprocessing scales and rotates the data. The nerf coordinate frame can be determined using the dataparser_transform.json in the output folder. This can be used to get to the coordinate system of the transforms.json file.

[Baptiste-AIST]

I guess you're talking about this function colmap_to_json() in colmap_utils.py But I don't see any scaling there, am I missing something somewhere else?

[Baptiste-AIST]

Okay, I found where the scaling happens https://github.com/nerfstudio-project/nerfstudio/blob/2b8d5a9e5c2dba4332a62434355631c13f6d9c96/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#L70

The centering and scaling are done in the nerfstudio dataparser, contrarily to instant-ngp where it is done in colmap2nerf.py script and are contained in directly in the transforms.json.

It is possible to use the save_dataparser_transform to save the transformation matrix used to center the cameras and the scaling. I'm not sure where it is possible to call this function, it could be great to have another argument in the command line to generate this file.

https://github.com/nerfstudio-project/nerfstudio/blob/c0464f8a9113636f877841302a0e48613cb662f8/nerfstudio/data/dataparsers/base_dataparser.py#L77

Edit: this file is automatically generated when training inside the output folder

[cdcseacave]

is there a way to export the point-cloud and/or mesh scaled/transformed back to the original coordinates?

[LHXhh]

@cdcseacave Hello, have you implemented it yet“is there a way to export the point-cloud and/or mesh scaled/transformed back to the original coordinates?”

[cdcseacave]

no

[dustier]

Hey, @cdcseacave @LHXhh

1. get transform matrix and scale from dataparser_transforms.json in output folder
2. scale back and transform back to origin coord system

mesh = open3d.io.read_triangle_mesh("/path_you_ns_export_to")
mesh.scale(1 / scale, center=(0,0,0))
mesh = mesh.transform(np.linalg.inv(transform_matrix))) # this is the mesh in origin coord system

[engrkhurramshabbir]

Hi, @dustier I

Hey, @cdcseacave @LHXhh

1. get transform matrix and scale from dataparser_transforms.json in output folder
2. scale back and transform back to origin coord system

mesh = open3d.io.read_triangle_mesh("/path_you_ns_export_to")
mesh.scale(1 / scale, center=(0,0,0))
mesh = mesh.transform(np.linalg.inv(transform_matrix))) # this is the mesh in origin coord system

Hi, I apply this transformation on the point cloud. But scale factor is not accurate. For example my object length is 100 meters but I am getting 64 meters for the same length.

[elenacliu]

has this problem been solved yet?

[kerrj]

Export scale should be working if you apply the dataparser transform, I've used it fairly extensively for robot related projects and the coordinates line up with the real world.

[elenacliu]

I'm not sure, but if I want to compare the ground truth depth and the depth generated by nerfacto (and there is transform and scale to the original c2w matrix), how should I align the depths?

If you scale the camera extrinsics, will the world coordinates remain the same (in the following equations, do the left hand sides equal)? I just cannot figure out the relation of the original depth and the output depth by nerfacto (depth and depth')

[Yaodizhao]

如果您应用数据解析器变换，导出比例应该可以工作，我已经将它广泛用于机器人相关项目，并且坐标与现实世界一致。 May I ask if I can convert the generated point cloud back to the original data, but there is a significant difference when I perform calculations. May I ask if you can give me some help

[chrisbin07]

Export scale should be working if you apply the dataparser transform, I've used it fairly extensively for robot related projects and the coordinates line up with the real world.

@kerrj I have been trying to use the dataparser_transforms.json file to rescale the point cloud (I'm more concerned about the actual scale of the objects), which should be as simple as dividing the coordinate values by the inverse of the 'scale' value in the json file, but it is not adding up.

am I missing something?

[cdcseacave]

I also tried to transform the scene back into the original coordinate system, and also had problems with the scale/rotation; the only way I was able to make it work is to disable the rotation transform when normalizing the scene at load time; scale and translation can still be applied, but no rotation as it does break the rendering of the splats when transformed back in original CS

[chrisbin07]

I also tried to transform the scene back into the original coordinate system, and also had problems with the scale/rotation; the only way I was able to make it work is to disable the rotation transform when normalizing the scene at load time; scale and translation can still be applied, but no rotation as it does break the rendering of the splats when transformed back in original CS

Was your problem specific to splatfacto? I'm trying to train a nerfacto model with a metashape dataset I processed, I haven't faced any problems with the rotation transform

[hanjoonwon]

Describe the bug Hi everyone, I'm not too sure if it's a bug, but I wanted to know how the generated point cloud or meshes are scaled when exporting. I tried to compare the sizes from instant-ngp, nerfstudio and COLMAP and they differ between each other (using the same transforms.json). Is there a place when training or exporting where things get scaled?

Thank you very much!

Expected behavior At least I was expecting nerfstudio to generate a similar scaling as COLMAP or instant-ngp

Screenshots

Marker size from instant-ngp marshing cubes mesh export

0.278130m

Of course the measures are not so precise, but we can see the order of magnitude difference.

have you solve this problem?

[hanjoonwon]

Export scale should be working if you apply the dataparser transform, I've used it fairly extensively for robot related projects and the coordinates line up with the real world.

How can i get real size mesh? I tried to #2924 but failed

[smart4654154]

Thank you very much for your work. My question seems to be related to this job. What is the meaning of the crop scale value in the new version of viewer? I found that the value of the crop scale cannot match the coordinate values of the world coordinate system. For example, when the crop scale is set to 4, 4, 4, it is approximately equivalent to a square with a side length of 40 in the world coordinate system What are the meanings of "crash max" and "crash min" in Viewer_legacy?

[brentyi]

For example, when the crop scale is set to 4, 4, 4, it is approximately equivalent to a square with a side length of 40 in the world coordinate system

How are you measuring the world coordinate system? "Crop scale" is indeed applied in the scale coordinate system discussed in this issue. This is computed roughly so the entire scene fits in a 1x1x1 box, which seems to match what you're seeing.

What are the meanings of "crash max" and "crash min" in Viewer_legacy?

The crop_min, crop_max define two corner of an axis-aligned bounding box. The units should be the same as the Crop scale units in the newer viewer.

[smart4654154]

Thank you for your reply. I added a point cloud to the viewer using the add_point_cloud() function in the viser package, which is located in the world coordinate system and can be adjusted in position. My measurement steps: 1.the drop scale is set to 4, 4, 4. 2.when my point cloud position is (20, 20, 20) or (-20, -20, -20), it coincides with the vertices of the bounding box. You can see it in the screenshot. This indicates that when the crop scale is set to 4, 4, 4, it is approximately equivalent to a square with a side length of 40 in the world coordinate system. I think the value of the crop scale in my presentation should follow the world coordinate system. In fact, the value of the crop scale is (4, 4, 4), which is not consistent with the world coordinate system. My question is, does the crop scale value not comply with the world coordinate system? (corp scale is used in new version of the viewer). What is the meaning of Crop_scale?

'crash max' and 'crash min' are 'crop_min', 'crop_max'.This is a spelling error

[brentyi]

This scale and how it's used might be interesting to you: https://github.com/nerfstudio-project/nerfstudio/blob/8e0c68754b2c440e2d83864fac586cddcac52dc4/nerfstudio/viewer/viewer.py#L51

Your [-20, -20, -20] should be divided by that ratio, which gets us to [-2, -2, -2]. That seems consistent with your 4x4x4 crop scale!

[alancneves]

Hey, @cdcseacave @LHXhh

1. get transform matrix and scale from dataparser_transforms.json in output folder
2. scale back and transform back to origin coord system

mesh = open3d.io.read_triangle_mesh("/path_you_ns_export_to")
mesh.scale(1 / scale, center=(0,0,0))
mesh = mesh.transform(np.linalg.inv(transform_matrix))) # this is the mesh in origin coord system

Worked like a charm for me!

[smart4654154]

Hey, @cdcseacave @LHXhh

1. get transform matrix and scale from dataparser_transforms.json in output folder
2. scale back and transform back to origin coord system

mesh = open3d.io.read_triangle_mesh("/path_you_ns_export_to")
mesh.scale(1 / scale, center=(0,0,0))
mesh = mesh.transform(np.linalg.inv(transform_matrix))) # this is the mesh in origin coord system

Worked like a charm for me! thank you Is this method useful for 3DGS? 3DGS is the result ply exported from nerfstudio's splatfacto. I think if the position of the point is bound to other parameters, this transformation may not be meaningful