DRAEM icon indicating copy to clipboard operation
DRAEM copied to clipboard

CUDA out of memory(test_DREAM.py)

Open PeterKim1 opened this issue 4 years ago • 2 comments

Hello.

I tried to run test_DREAM.py, but CUDA out of memory occur. (I use 15.90 GiB memory)

I don't know well about all of your codes, but when i check GPU memory, memory usage continue to increase.

I think some variables need to be allocated at CPU. (by using .cpu().detach())


After many attempts, I know where i need to change.

https://github.com/VitjanZ/DRAEM/blob/main/test_DRAEM.py#L96:L99

Before(Original codes)

display_images[cnt_display] = gray_rec[0] display_gt_images[cnt_display] = gray_batch[0] display_out_masks[cnt_display] = t_mask[0] display_in_masks[cnt_display] = true_mask[0]

After(changed codes)

display_images[cnt_display] = gray_rec[0].cpu().detach() display_gt_images[cnt_display] = gray_batch[0].cpu().detach() display_out_masks[cnt_display] = t_mask[0].cpu().detach() display_in_masks[cnt_display] = true_mask[0].cpu().detach()

PeterKim1 avatar Oct 25 '21 09:10 PeterKim1

Thanks, this helps a lot.

Prophet-C avatar Jan 23 '22 17:01 Prophet-C

or adding with torch.no_grad(): before testing (line 77) also works. and i removed the "display" part cause it's not used in the code, i guess it's for visualization.

ghost avatar Apr 02 '22 08:04 ghost