neus-facto-angelo takes huge amount of VRAM

[juuso-oskari]

Hi,

I have a A4500 laptop GPU with 16GB of VRAM. I'm not able to run the neus-facto-angelo however:

ns-train neus-facto-angelo --pipeline.model.sdf-field.inside-outside False --vis wandb --experiment-name test sdfstudio-data --data data/sdfstudio-demo-data/dtu-scan65

    native_ctx, output = native_tcnn_module.fwd(input, params)
RuntimeError: /tmp/pip-req-build-7pl_khv_/include/tiny-cuda-nn/gpu_memory.h:584 cuMemCreate(&m_handles.back(), n_bytes_to_allocate, &prop, 0) failed: CUDA_ERROR_OUT_OF_MEMORY

How much VRAM does this method require or should there be some additional input arguments?

[niujinshuchong]

Hi, you can try to reduce the learnable parameters of the model and also reduce batch size.

[juuso-oskari]

Hi @niujinshuchong, I was able to get the model running by reducing the training_num_rays_per_batch considerably. But it still crashes for the evaluation with: RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

I looked for this and its also probably caused my out of memory. How can it run out of memory when the eval_num_rays_per_batch is much lower than training_num_rays_per_batch for me? I'm running the evaluation for all the images.