Out of VRAM for --vram_0 with 128x128 resolution.

[ondrejbiza]

Description

Hi, I tried running dreamfusion with higher resolution, as suggested in the README: python main.py --text "a hamburger" --workspace trial -O --vram_O --w 300 --h 300

However, I get a CUDA out of memory error on my RTX 2080 Ti with 11GB of RAM.

OutOfMemoryError: CUDA out of memory. Tried to allocate 444.00 MiB (GPU 0; 10.76 GiB total capacity; 8.75 GiB already allocated; 431.00 MiB free; 9.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I found that this error happens even with: python main.py --text "a hamburger" --workspace trial -O --vram_O --w 128 --h 128 I can only run it successfully with 64x64 images.

I tested both PyTorch 1.13.1 and PyTorch 2.0 with CUDA 11.7, but got the same problem. I cloned the repo today (4/17) so perhaps a recent change broke this?

P.S. Thanks for the great repo!

Steps to Reproduce

python main.py --text "a hamburger" --workspace trial -O --vram_O --w 300 --h 300

Expected Behavior

python main.py --text "a hamburger" --workspace trial -O --vram_O --w 300 --h 300 should not run out of VRAM per README.

Environment

Ubuntu 20.04, PyTorch 1.13.1, CUDA 11.7 Ubuntu 20.04, PyTorch 2.0, CUDA 11.7 (I installed bash scripts/install_ext.sh)

[ondrejbiza]

Same thing happens in colab with Tesla T4: main.py -O --vram_O --text 'a DSLR photo of a mug' --workspace trial4 --iters 5000 --lr 0.001 --w 300 --h 300 --seed 0 --lambda_entropy 0.0001 --ckpt latest --save_mesh --max_steps 512

CUDA out of memory. Tried to allocate 774.00 MiB (GPU 0; 14.75 GiB total capacity; 12.29 GiB already allocated; 498.81 MiB free; 12.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[ashawkey]

@ondrejbiza Hi, I have observed this too. I guess it's the removal of normal_net and using of finite difference normal that causes the increased memory usage. I'm still looking for a better solution, but for now you can train NeRF at 64x64 and finetune DMTet at 512x512, which still gives high resolution results.