Consultation on CUDA Out-of-Memory Issue When Running the Code with TCGA - BRCA Dataset

[blz822]

Dear author,

I'm using your code and have encountered some issues. Here is some information about my setup and the output I got:

Environment Information

• I'm using a single NVIDIA GeForce RTX 3090 GPU with 24GB of memory.
• The dataset I'm using is TCGA - BRCA.

Output

The size of train dataset: 804
The size of val dataset: 95
The size of test dataset: 98
use encoder_decoder: default
Training:  33%|██████████████████████████████████████████▏                                                                                    | 267/804 [00:31<01:03,  8.4
Traceback (most recent call last):
  File "main.py", line 198, in <module>
    mp.spawn(main,
  File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/mnt/data0/BLZ/Wsi-Caption-master/main.py", line 181, in main
    trainer.train(local_rank)
  File "/mnt/data0/BLZ/Wsi-Caption-master/modules/trainer.py", line 58, in train
    result = self._train_epoch(epoch)
  File "/mnt/data0/BLZ/Wsi-Caption-master/modules/trainer.py", line 238, in _train_epoch
    loss.backward()
  File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.55 GiB (GPU 0; 23.69 GiB total capacity; 19.42 GiB already allocated; 923.69 MiB free; 21.33 GiB reserved in total brch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

It seems that the code is running out of CUDA memory. I've checked the error message, but I'm not sure how to fix it properly. Could you please give me some advice on how to modify the code to avoid this CUDA out - of - memory issue?

Thank you very much!

[anhtienng]

I think the model is big itself, I'm using A6000 with 44GB is allocated for training !

[jasonlightx]

I use the same equipment as you, but I happen to be able to run the training smoothly, but the results were not satisfactory during testing. Are you also in the same situation

[anhtienng]

I got the similar situation too !

[jasonlightx]

I got the similar situation too !

Are you also generating a bunch of repetitive and meaningless reports？