Shixin Jiang comments

Repositories
Issues
Comments

Results 3 comments of


                                            Shixin Jiang

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

> I am facing the above error while running the training script. Can someone let me know how to solve it? > > My GPU specifications are- ![image](https://user-images.githubusercontent.com/116613755/264011532-ab89e3ce-368d-44ad-a599-9bfd67d07ded.png) > >...

Actor grad Norms are always NaN (GRPO Training)

same after 80 steps for multi-turn grpo training on qwen2.5-instruct-3b

Actor grad Norms are always NaN (GRPO Training)

> same after 80 steps for multi-turn grpo training on qwen2.5-instruct-3b after try warmup=0.285, the phenomenon is alleviated