dust3r icon indicating copy to clipboard operation
dust3r copied to clipboard

Getting torch.cuda.OutOfMemoryError using more than 16 images

Open rowellz opened this issue 11 months ago • 1 comments

Firstly, congrats to all the folks at Naver for their awesome accomplishments with Dust3r. It was very straight forward getting dust3r up and running, but I discovered i run into torch.cuda.OutOfMemoryError(s) when i try to process more than 16 images at once. I am running a rtx 3060 12GB, was wondering if anyone may know what I can do to resolve or debug an issue like this? I am running dust3r via docker-compose with PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128. Here is the full error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.11 GiB (GPU 0; 11.76 GiB total capacity; 9.84 GiB already allocated; 932.69 MiB free; 10.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Any help or insight would be much appreciated. Again, thanks to the folks at Naver for their awesome work and releasing it!

rowellz avatar Mar 06 '24 18:03 rowellz

Hi,

You can refer to the issue #1.

Some suggestions may help:

  1. Set device='cpu' below, the inference time will be longer. https://github.com/naver/dust3r/blob/7744dbae4475ebda78cd5cb9e6e60aa1eaefa5a4/demo.py#L136

  2. Use the "swin" or "oneref" method to make fewer pairs at global alignment as mentioned https://github.com/naver/dust3r/issues/1#issuecomment-1973308527 image

ffrivera0 avatar Mar 06 '24 22:03 ffrivera0