How to use input equirectangular images for training?

[HungNgoCT]

Hi,

I saw in a PR that nerfstudio support equirectangular images as this link: https://github.com/nerfstudio-project/nerfstudio/pull/1024

Can anyone guide me that how to use equirectagular images for training? I understood that I will use ns-process-data for for images and pose generation, then use ns-train for training. I checked ns-process-data but don't know how to use it with input equirectangular images as the image below.

Highly appreciate if anyone can help.

Hung

[tancik]

We currently don't have a pipeline for inputing equirectangular images. This is because the method used to find the poses (COLMAP) does not support equirectangular. There was some work on integrating OpenSFM which does support 360, but the PR has become stale - https://github.com/nerfstudio-project/nerfstudio/pull/969

[machenmusik]

For clarity - I see https://github.com/nerfstudio-project/nerfstudio/pull/966 which claims it adds equirectangular image support, but it sounds as though there is no ns-process-data method that uses that yet - is that right?

[mgupta70]

I have been experimenting with 360 videos. I split them into multiple overlapping perspective images and register those in colmap. Registration was successful but the 3D reconstruction with Instant NGP is not good. Although, The dense 3D reconstruction in Colmap is much better. I tried the same set of photos in RealityCapture and 3d reconstruction is comparatively of a very high quality.

Can you think of a reason why it could happen?

[tancik]

How does it looks with the nerfacto method?

[mgupta70]

I did not try with nerfacto. Will try it and post it here. But any guesses @tancik as to why this could be happening ?

[tancik]

Maybe you don't have enough images. I'd also like to see a viewpoint from inside the room.

[mgupta70]

I have attached an equirectangular image and two of the perspective images from it. I have 12 equirectangular images and 120 perspective images.

[tancik]

Can you post a screenshot of a nerf reconstruction from inside the room. Right away though, I can tell you that the helmet in the view will cause issues. It will likely lead to lots of floating geometry around the locations where the images were captured

[mgupta70]

It does create a lot of floating geometry. I was wondering why and now what you said makes sense. I am attaching a small video and 2 snapshots from inside the reconstructed scene

https://user-images.githubusercontent.com/110718346/219147344-a7fca84a-5aab-4d42-b2fe-b34bd4c2e919.mp4

[tancik]

It looks like you aren't using nerfstudio.

[mgupta70]

Will do it 🤞

[mgupta70]

I used nerfacto to train this time. I split equirectangular images [5760x2880] into 8 perspective images [1200x1200] (from meshlab) such that helmet is not present. I am attaching a equirectangular image and 2 of its perspective images. The reconstruction was not good again. I am attaching snapshots of it however, path of cameras and camera poses look accurate because I captured the data.

https://user-images.githubusercontent.com/110718346/219466883-a00c7dbc-20f7-4470-805d-b605ba3a60a6.mp4

[tancik]

It is very likely that the poses are incorrect. These are the artifacts that you would expect in this case. Did you run your images through ns-process-data?

[mgupta70]

yes, I did it with ns-process-data which then ran Colmap 3.8. I cannot understand why is this happening?

I have 2 queries in this regard:

1. Can it be related to the scale of cube in which we render? (Like aabb_scale in NGP)?
2. Is it a good idea to remove those images which have less information. I mean For ex: if an image only has plain white wall in background and only a small portion in that image is occupied by another object.

Thank you @tancik for replying

[tancik]

1. The nerfacto model autoscales the scene so you don't need to worry about aabb_scale.
2. Including that data is fine.

Can you send me your processed data.

[mgupta70]

I am sharing it on drive. I am sending you an email for this.

[tancik]

I tested the data, it also doesn't work for me, Im not sure why. We will be releasing some 360 processing tools soon, they might help with this data.

[mgupta70]

oh :( Thanks for looking. I will be eagerly waiting for the updates. Meanwhile, I will play around with other parameters like Sampler and matching method to be exhaustive Can you point me to some areas which I could explore? For now, I am trying preprocessing to filter out blurry images using Laplace of variance and performing some techniques like Histogram Equalization for enhancing exposure.

[tancik]

https://user-images.githubusercontent.com/3310961/219978377-f0136119-94b8-4a80-843a-0140df51204e.mp4

There was a bug introduced in nerfstudio 0.1.17 A patch will be coming soon. Alternatively you can use 0.1.16 in the meantime.

[mgupta70]

Woah!!! This looks good. Thanks a lot. I will shift to 0.1.16.

[tancik]

v0.1.18 is out, use that instead.