lambda_depth=0.0 in arguments/__init__.py and the role of depth loss during training

[guijuzhejiang]

Hi, I noticed that in arguments/init.py, the default value of self.lambda_depth is set to 0.0. If I understand correctly, this means the depth_loss is not computed during training, right?

If depth_loss is not computed, does that mean the ground truth depth generated by MiDaS is not actually used? My question is: is it intended that depth_loss is excluded during training? Or, under what circumstances should depth_loss be enabled?

Could you please clarify the purpose of setting lambda_depth=0.0 and whether we need to adjust this value to incorporate depth loss in the training process?

Thanks in advance for your help!

[amritamaz]

Hi, thanks for your question. The depth loss is not used during training, but the MiDaS depth is used to help initialize the COLMAP points (see this section of the training script, or Appendix A.5 in the paper. This is enabled by setting depth_init: True in the configs/dynerf.yaml.

[guijuzhejiang]

Thanks, I experimented with replacing MiDaS with Depth Anything V2 for depth_init, expecting better reconstruction due to DA2's superior depth estimation. However, **final PSNR showed no improvement. Does depth initialization quality significantly impact final metrics in your experience?

[amritamaz]

For scenes where the initial COLMAP point cloud is sparse, we found that using depth estimation helped with reconstructing that first frame. I don't believe we did experiments comparing different depth initialization methods, though. If you are trying to supplement a decent initial point cloud, the depth_init method may not help much.