Misalignment Issue with Custom Data from Isaac Sim Camera, model-free

[shingarey]

Firstly, I want to express my gratitude for releasing this library.

I'm currently facing an issue with misalignment by inference. I recorded custom data from an Isaac Sim camera, gathering both image and depth data from the ROS2 topics. I then used bundlesdf to create a mesh and ran python bundlesdf/run_nerf.py on the data.

However, when I started run_demo.py, I noticed the misalignment. I'm not sure why this is happening. Have you encountered this issue before or do you have any idea why this might be occurring?

Mesh from bundlesdf:

Video of run_demo.py:

https://github.com/NVlabs/FoundationPose/assets/22908950/4795ddb1-0484-425e-ac66-2db7621c2655

[shingarey]

It turns to be out, that I encountered a problem when using the model generated by run_nerf.py located in model/model.obj. For some reason, it doesn't seem to be well-reconstructed:

However, when I switched to the mesh generated by bundlesdf (see post above), the issue was resolved.

@wenbowen123, do you have any insights as to why the model in model/model.obj appears so strange?

[wenbowen123]

Hi, when you run the run_nerf.py from this repo, how did you provide the camera poses? Is it from object to camera or opposite?

[wenbowen123]

And have you tried with the model-free pipeline on LINEMOD/YCB-Video dataset to get reasonable results? The steps are provided in readme.

[shingarey]

Hi, when you run the run_nerf.py from this repo, how did you provide the camera poses? Is it from object to camera or opposite?

Hi, I used the poses provided by BundleSDF, which are defined as obj_in_cam. So, the inverse of these poses should be used in cam_in_obj, correct?

Could you suggest the best method to define cam_in_obj for each image without utilizing the BundleSDF library? Thank you.

[wenbowen123]

yes they are the inverse.

you can use other SLAM methods to get the camera poses, if you dont want to run BundleSDF. However, BundleSDF has been tested with FoundationPose many times and turns out to work the best. Is there any reason you dont want to use BundleSDF?

[shingarey]

There's no particular reason for not wanting to use it. It just took me some time to install, and I was wondering if there might be a more lightweight alternative. However, once installed, it functions well.

[ssapsu]

@shingarey Hello shingarey,

I'm curious about how you've managed to use ROS2 with conda? If it's okay with you, may I ask about it?