Training Instability

[ktertikas]

Hi @donydchen , thanks for your amazing work!

I wanted to ask, have you experienced any instability during training on the DL3DV data? I am getting nan values at a random point during training (see attached training plots below). On a similar note, I suspect that there might be some issues with the processed data (found a scene with truncated images, another with issues in the rotation matrices). Could you please send over the list of DL3DV scene ids that are kept during training?

Best, Konstantinos