Jiahao Li

icon indicating a website link https://liplus.me icon indicating an email [email protected]

icon company @bytedance icon location Beijing, China icon location LLM Infra @ ByteDance Seed | B.Eng in Computer Science @ Tsinghua University

Results 13 issues of Jiahao Li

Fix labels & eos_token for SFT

Fixed two issues: * Padding should be ignored in training. Their labels should be set to `-100` for `CrossEntropyLoss` to ignore them. * Append correct `eos_token` to the response text....

Enable overlap_comm for better performance

Enable overlap of backward computation and gradient all-reduce. This produces 1.05x end-to-end speedup in SFT training with my settings. See also https://github.com/microsoft/DeepSpeed/pull/4887.

Add manualOrderTime for exerciseOptions

See https://interactivebrokers.github.io/tws-api/options.html for the new arg `manualOrderTime`