Any hint code on calculating SDF, do we need to rotate it first?

[wangzheallen]

Any hint code on calculating SDF and do we need to translate it to camera coordinate and calculate? or we can use the world-coordinate one for fitting?

[dimtzionas]

Can you please clarify further your question? Do you want to compute the SDF for a new 3D scene scan, or use the existing ones?

[wangzheallen]

I want to calculate SDF for a new 3D scene. I refer to the code of https://github.com/marian42/mesh_to_sdf From the code I can get points & SDF & voxelization resolution, which corresponds to vicon_nn.npy & vicon.sdf.npy & dim in vicon.npy

However, I cannot get the max&min in vicon.npy as well as vicon_normal.npy

The big problem is how to compute SDF for a new 3D scene scan (or obj file), small problem is I tried but still cannot get the max&min & normal.

Thanks!

[dimtzionas]

For getting the SDF of a new scene:

• You need to define a 3D voxel grid that covers your scene (please see the text above Eq.6 in PROX paper). It is recommended that the voxel grid is a bit bigger than the scene (i.e. covering some space around the scene), so that you can penalize the current human estimation when it is bad, e.g. when people appear to intersect with walls or even get outside them or penetrate the floor. Please have in mind that it practically helps if the room walls and floor are parallel to the XYZ axes/planes of your world coordinate system, so you might need to rotate a bit your scene/room meshes.
• Then, for each voxel you need to compute the closest distance of its center to the scene mesh, as well as compute if its center is inside some object or lying outside in free space (please see Eq.6 in PROX paper). Due to licensing issues the exact code for this is not online, but this might be revisited if issues are resolved. In any case, you can use the CGAL fast distance and intersection computation package to help you speed things up a bit if your meshes are of high resolution. If your scene is of high resolution, even point-to-point distances might be good enough, depending on your application. However, usually we recommend point (voxel center) to surface (scene triangles) distances (often flat surfaces like walls and floors have much bigger triangles).
• The repo that you reference probably does the above steps, but chances are that they do this for a tight voxel grid around your mesh. We suggest you to track the code down and relax this a bit (think of this like "padding" a 2D image).

After this, as you say, you have things in two different systems. What your camera observes is in the camera coordinate system, and the scene mesh and SDF is in the scene/world coordinate system. This is like defining a virtual camera that "sees" the scene mesh in the same way as the real camera "sees" the real scene. For this you need to find a rigid transformation (rotation and translation) from one coordinate system to the other, i.e. perform extrinsic calibration. For this you need a minimum of three correspondences. Please see Section 3.2 in the PROX paper. For this calibration, we capture with the camera 1 RGB/depth image without any human in it. A human annotator clicks on 3 (2D) salient points in the RGB/depth image (Kinect captures aligned RGB and depth images) and the corresponding 3 (3D) points on the scene mesh (be aware of some holes that the mesh might have). These 3 correspondences are the constraints you need to compute the transformation. Please have in mind that there might be some noise in the annotation, so this transformation will be a bit rough, but it gets you into the ballpark of aligning the captured depth image (it "sees" a part of the scene, i.e. the scene mesh) to the 3D scene mesh. We use this to initialize an ICP stage, that refines this alignment with iterative correspondence estimation and optimization. For the steps of this paragraph we used the Open3D library, which was very handy, so we totally recommend it.

Unfortunately I am not fully sure I understand the problem you describe for max&min and normal. If you need the max/min XYZ coordinates of the mesh to define your "relaxed" voxel grid, then this is simple with numpy operations (on mesh vertex coordinates). If you need to reconstruct the normals of the scene mesh, you can use tools like MeshLab or Open3D. Please be aware that sometimes the output normals of such tools might need flipping (this was the case in the past with the PCL library for example), i.e. check out the result with a 3D viewer before using it. In our case we were lucky to use the Structure camera and its Skanect software for reconstructing the scenes, this software outputs a mesh together with clean normals out of the box.

I hope the above clarify things :)

[thusiyuan]

Hi,

Can we directly use the preprocessing code in DeepSDF https://github.com/facebookresearch/DeepSDF/blob/master/preprocess_data.py to get the SDF from the given 3D mesh?

Thank you!

[mohamedhassanmus]

Hi! My apologies for the late reply. Unfortunately, the SDF computation code can't be made public yet.

I didn't look at the DeepSDF code but this library might be useful https://github.com/marian42/mesh_to_sdf. You should pay attention to use the sign_method='normals'. The other method which computes the sign based on the depth maps will NOT work for PROX scenes. The only issue is that it is a bit slow because it doesn't use any accelerated structure.

Another option would be to look at PyTorch3D. They have some function for computing the KNN on point clouds https://pytorch3d.readthedocs.io/en/latest/modules/ops.html#pytorch3d.ops.knn_points. I don't see a function to compute point2surface distance but take a look maybe I missed. What you need is a function to compute the distance between a point and a surface. You could approximate that by sampling densely on the scene surface and then use the KNN function above to compute the distance between your query points and the surface points. The other problem would be to calculate the sing of the distance. For this, you need the normals of the surface. You could look at the implementation in the mesh_to_sdf repo above and try to replicate it in PyTorch. I think it should be possible.

[thusiyuan]

Thank you for your answer!

[GuHuangAI]

Thank you for your answer!

hello, where is the "models" directory under the dataset root? Could you help me?