[FEATURE]: please add 8-bits NLP model training mode.

[yynil]

Describe the feature

Currently FP16 support can only make it possible for training models smaller than 2B in one graphic card with 24gb ram. However the main stream useful model size is around 6~7B, I can't even load the model in FP16 mode. If I loaded in 8bits, the training will fail due to datatype mismatched.

Would you please add the 8bits model training in the near future?

[JThh]

After resolving the licensing issue, we may be able to provide users with some pre-trained weights. You may deploy and do inference with 8 bits then. But for now, your advice is appreciated and we will consider it.

[binmakeswell]

Hi @yynil We have add How to train with limited resources https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat#faq For 8bits training, we are not currently making it a high priority task due to the potential for serious convergence issues. Thanks.