gaussian-opacity-fields Adding Depth Supervision causes bad meshes and unstable results

Hello,

I am working on a fork to add depth supervision to GOF to see if it improves the quality of the extracted meshes, particularly in areas with limited image coverage. I am using the AbandonedFactory scene from the TartanAir dataset, which contains ground truth depth images alongside each RGB image.

However, doing so seems to cause the training process to become extremely unstable and results in completely unusable meshes. Here are the results from two identical runs with the same code:

Run 1

Results	Epoch 1000	Epoch 7000	Epoch 15000	Epoch 30000	Ground Truth
Depth
RGB

Loss Statistics	Mesh Results (outside)	Mesh results (inside)

Run 2

Results	Epoch 1000	Epoch 7000	Epoch 15000	Epoch 30000	Ground Truth
Depth
RGB

Loss Statistics	Mesh Results (outside)	Mesh results (inside)

Would you have any insights as to what might be causing this complete degredation in performance? In particular I notice that there is always a large spike in loss at 15000 epochs where the number of Gaussians in the scene stops increasing, and in some of the depth images there appears to be almost cyclic depth patterns (such as around the bricks) reminiscent of what happens when cutting the bit depth of an image (but I know for sure I'm normalizing the images beforehand).

I took the code for extracting the depth from extract_mesh_tsdf.py and apply a simple loss while masking out the extreme values:

(see the full code at https://github.com/Soldann/gaussian-opacity-fields/blob/7b2dcc565ddd1786685ea00f8f7179176cbb989f/train.py#L196)

The only other idea I have is perhaps the units are off as the depth image provided by TartanAir is in metres, but if I'm not mistaken the depth map produced by the renderer should also be in metres? Any ideas or guidance would be much appreciated.