facebookresearch
CUDA out of memory.
Thank you for bringing us such an excellent work — it's been very helpful to our research. However, I would like to ask a question: I downloaded the ViT-7B weights and used the corresponding configuration file, but even with batch_size = 1, I encountered the following error:
CUDA out of memory. Tried to allocate 96.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 78.75 MiB is free. Including non-PyTorch memory, this process has 79.17 GiB memory in use. Of the allocated memory 77.97 GiB is allocated by PyTorch, and 253.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management: https://pytorch.org/docs/stable/notes/cuda.html#environment-variables
My GPU is an NVIDIA A100 (80GB), and I'm surprised that even with batch size = 1 this error still occurs. Could you please advise whether this is expected behavior for ViT-7B in your setup, or if there are any adjustments recommended (e.g., precision settings, input resolution, gradient checkpointing, etc.)?
