CUDA out of memory after 300 iterations on A40(40GB)

[junaidwahid]

Hey, after doing validation at 300, it gives a CUDA out-of-memory error. This error persists when I try to resume training from the 300th iteration. I am training Stable-Zero123.

[DSaurus]

Hi, @junaidwahid

You could consider changing the batch_size to [12,8,2] or use threestudio-stable-nerf-renderer extension.

  random_camera: # threestudio/data/uncond.py -> RandomCameraDataModuleConfig
    height: [64, 128, 256]
    width: [64, 128, 256]
    batch_size: [12, 8, 4]
    resolution_milestones: [200, 300]
    eval_height: 512
    eval_width: 512