ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard
verified publisher icon hpcaitech

[FEATURE]: support for gradient clipping by value

Open haofanwang opened this issue 2 years ago • 1 comments

Describe the feature

Is there any plan for supporting clip_grad_value in Pytorch?

haofanwang avatar Jan 11 '23 18:01 haofanwang

Can you post more information about the PyTorch clip_grad_value? How to use it? Maybe some code snippet is more helpful.

feifeibear avatar Jan 13 '23 06:01 feifeibear

Hi @haofanwang , its torch version shall work well with ColossalAI. In addition, we have our own clip_grad_norm in colossalai.utils.common for the correctness of model parallelism.

kurisusnowdeng avatar Jan 30 '23 04:01 kurisusnowdeng