Zijian Hu

icon indicating a website link www.zijianhu.com icon indicating an email [email protected]

icon company @scaleapi icon location San Francisco, CA, California icon location ML research at Scale AI.

Results 1 issues of Zijian Hu

Gradient norm clipping with pipeline parallelism (PP)

5 comment

Dear torchtitan team, I have a question regarding gradient norm clipping when using pipeline parallelism (PP) potentially combined with `FSDP/DP/TP`. For simplicity, let's assume each process/GPU has single PP stage....

bug