ColossalAI-Examples icon indicating copy to clipboard operation
ColossalAI-Examples copied to clipboard

Published 20 hours ago • verified publisher icon hpcaitech

grad is none when run gpt2 with pipeline parallelism only

Open lin88lin8850 opened this issue 2 years ago • 0 comments

🐛 Describe the bug

image

Environment

torch==1.12.0a0+8a1a93a num_gpu=4 pipeline=4 model = gpt2-small

lin88lin8850 avatar Dec 22 '22 06:12 lin88lin8850