threestudio icon indicating copy to clipboard operation
threestudio copied to clipboard

Published 20 hours ago • verified publisher icon threestudio-project

InstructNerf2Nerf custom capture

Open yankeesong opened this issue 2 years ago • 15 comments

I am running IN2N on a custom capture. I first used NerfStudio to pre-process the data as illustrated here. However when used the processed data in IN2N pipeline:

  1. First I got parsing error (which I fixed)
  2. Then the training completely messed up, giving "[WARNING] Empty rays_indices!" all the time.

Has anyone tried running IN2N on a custom capture and got similar errors?

yankeesong avatar Jul 31 '23 08:07 yankeesong

Hi, @yankeesong. Could you please share your data? I will try to help you figure out the issues.

DSaurus avatar Aug 01 '23 15:08 DSaurus

Edited

yankeesong avatar Aug 02 '23 03:08 yankeesong

https://drive.google.com/drive/folders/1v4MLNoSwxvSlWb26xvjxeoHpgjhi_s-s I also got the same warning when using the bear images provided by IN2N.

yuanzhi-zhu avatar Aug 02 '23 11:08 yuanzhi-zhu

@yankeesong Can you reconstruct your scene successfully with NeRFStudio?

DSaurus avatar Aug 03 '23 16:08 DSaurus

Yes I can. in2n from NeRFStudio also worked.

yankeesong avatar Aug 03 '23 17:08 yankeesong

Here are some issues I've found in the current implementation:

Key problem:

  • The grid_prune in nerfacc will prune all samples if the scene is not initialized properly, resulting in "Empty rays" warning. To resolve this, we should set grid_prune to false. Other issues:

The L1 loss appears to be too small. It should be calculated with a sum reduction. It would be better to use importance sampling to save memory and improve reconstruction quality when grid_prune is false.

DSaurus avatar Aug 04 '23 04:08 DSaurus

Thanks so much! Which branch is this modification in, or are you planning to push this modification?

yankeesong avatar Aug 04 '23 06:08 yankeesong

Hi, I'm also facing the same problem. How did you solve it eventually ? Many thanks

YScheung avatar Aug 23 '23 06:08 YScheung

I'm also facing the same problem. How did you solve it eventually? Thanks

xutanxing avatar Oct 06 '23 06:10 xutanxing

I'm also facing the same problem. How did you solve it eventually? Thanks !!

hrz2000 avatar Oct 25 '23 07:10 hrz2000

@yankeesong Have you solved this problem? Can you please tell us what to do？

hrz2000 avatar Oct 29 '23 16:10 hrz2000

Hi all, Sorry for the late reply. I eventually used NeRFStudio's InstructNeRF2NeRF implementation (since I was pressed for time). I haven't tried to implemented @DSaurus suggestions yet. I guess I should leave this issue open.

yankeesong avatar Oct 29 '23 19:10 yankeesong

@yankeesong Thank you very much for your reply. I'm sorry to bother you again. I'm looking for the same solution as you, but I also have a problem with in2n implemented by nerfstudio. Have you ever encountered this？https://github.com/ayaanzhaque/instruct-nerf2nerf/issues/60

hrz2000 avatar Oct 30 '23 02:10 hrz2000

Not that I recall (this was a while ago though). Hope you resolve this eventually!

yankeesong avatar Oct 30 '23 05:10 yankeesong