MAGICS-LAB
I always encounter this error during the fine-tuning evaluation phase
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.44 GiB. GPU 0 has a total capacity of 10.58 GiB of which 4.44 GiB is free. Including non-PyTorch memory, this process has 6.14 GiB memory in use. Of the allocated memory 5.77 GiB is allocated by PyTorch, and 178.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
I always encounter this error during the fine-tuning evaluation phase.Even though I've drastically reduced the amount of data and batch size, I still report this error at pretty much the same time. As far as I can tell, memory usage starts to creep up at a certain node until it overflows, and I've tried a lot of things that don't work. My graphics card is 2080TI, CUDA 12.3.The strangest thing is that the memory is fixed during fine-tuning training, while it overflows during evaluation, even though the evaluation file is only 50MB
I solved the problem for now by making the estimate data extremely small
Hey zlw1747832053,
I also encountered this error, but found this thread to be helpful. It appears that there is some memory leakage when transformer.Trainer computes metrics on the logits tensor so it is recommended to do any preprocessing of logits in the function preprocess_logits_for_metrics before they are sent to compute_metrics. I implemented this fix in this pull request if you would like to check it out!
