Yushun Zhang

icon indicating a website link https://zyushun.github.io/

icon location PhD student at the Chinese University of Hong Kong, Shenzhen, China https://zyushun.github.io/

Results 3 issues of Yushun Zhang

A potential bug in multi-GPU training

Hi, I found the following strange phenomena when running tiny llama pretraining. 1. When using multiple GPUs, I got **completely different results** when **running the same code twice**. Further, many...

A potential bug for multi-GPU training

5 comment

Hi, I found the following strange phenomena when running your code for tinyllama pretraining. 1. When using multiple GPUs, I got **completely different results** when **running the same code twice**....

IndexError: tuple index out of range

Hi Jiawei, I was trying Galore on TinyLlama-1B using the codebase https://github.com/jzhang38/TinyLlama on 4* A800-80GB. I encounter the following error: ``` [rank1]: optimizer.step() [rank1]: File "/mntcephfs/lab_data/zhangyushun/anaconda/tinyllama/lib/python3.10/site-packages/lightning/fabric/wrappers.py", line 74, in step...