LMSCNet icon indicating copy to clipboard operation
LMSCNet copied to clipboard

RuntimeError: CUDA out of memory

Open hitbuyi opened this issue 1 year ago • 0 comments

(pt110) hitbuyi@hitbuyi-Dell-G15-5511:~/AD_Projects/Pytorch_Project/LMSCNet$ python LMSCNet/train.py --cfg SSC_configs/examples/LMSCNet.yaml --dset_root ./dataset_kitti
2024-07-23 21:04:29,065 -- ============ Training routine: "SSC_configs/examples/LMSCNet.yaml" ============

2024-07-23 21:04:29,861 -- => Loading network architecture...
2024-07-23 21:04:29,926 -- => Loading optimizer...
2024-07-23 21:04:29,927 -- => No checkpoint. Initializing model from scratch
2024-07-23 21:04:35,266 -- => =============== Epoch [1/80] ===============
2024-07-23 21:04:35,266 -- => Reminder - Output of routine on ../SSC_out/LMSCNet_SemanticKITTI_0723_203809
/home/hitbuyi/.conda/envs/pt110/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:247: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
  warnings.warn("To get the last learning rate computed by the scheduler, "
2024-07-23 21:04:35,266 -- => Learning rate: 0.001
Traceback (most recent call last):
  File "LMSCNet/train.py", line 333, in <module>
    main()
  File "LMSCNet/train.py", line 319, in main
    best_record = train(model, optimizer, scheduler, dataset, _cfg, epoch, logger, tbwriter)
  File "LMSCNet/train.py", line 107, in train
    loss['total'].backward()
  File "/home/hitbuyi/.conda/envs/pt110/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/hitbuyi/.conda/envs/pt110/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 5.79 GiB total capacity; 1.88 GiB already allocated; 1.52 GiB free; 1.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

How much the CUDA memory is required to run this project?

hitbuyi avatar Jul 23 '24 13:07 hitbuyi