gm-kns comments

Repositories
Issues
Comments

Results 2 comments of


                                            gm-kns

Getting an Out of Memory (OOM) error when training a 7B model on 4x A100 80G GPUs using GRPO.

I am getting the same issue (on a different dataset), and it is CPU related. At some point, there is a memory spike, and training can go for 0.5 or...

Out Of Memory

same issue. There seems to be a memory leak, the training goes well until a huge spike, not even when a checkpoint needs to be saved.