dust3r
dust3r copied to clipboard
Getting torch.cuda.OutOfMemoryError using more than 16 images
Firstly, congrats to all the folks at Naver for their awesome accomplishments with Dust3r. It was very straight forward getting dust3r up and running, but I discovered i run into torch.cuda.OutOfMemoryError(s) when i try to process more than 16 images at once. I am running a rtx 3060 12GB, was wondering if anyone may know what I can do to resolve or debug an issue like this? I am running dust3r via docker-compose with PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
. Here is the full error:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.11 GiB (GPU 0; 11.76 GiB total capacity; 9.84 GiB already allocated; 932.69 MiB free; 10.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Any help or insight would be much appreciated. Again, thanks to the folks at Naver for their awesome work and releasing it!
Hi,
You can refer to the issue #1.
Some suggestions may help:
-
Set
device='cpu'
below, the inference time will be longer. https://github.com/naver/dust3r/blob/7744dbae4475ebda78cd5cb9e6e60aa1eaefa5a4/demo.py#L136 -
Use the "swin" or "oneref" method to make fewer pairs at global alignment as mentioned https://github.com/naver/dust3r/issues/1#issuecomment-1973308527