binmakeswell

icon indicating an email [email protected]

icon company @hpcaitech icon location Singapore icon location Stay Hungry, Stay Foolish.

Results 359 comments of binmakeswell

[FEATURE]: train seq2seq model

@Sahajtomar This is just an example, welcome to try other models and submit PRs. For example #3567 Thanks.

[BUG]: assert self.l2_norm is None, "you are calculating the l2 norm twice"

> > Bot detected the issue body's language is not English, translate it automatically. πŸ‘―πŸ‘­πŸ»πŸ§‘β€πŸ€β€πŸ§‘πŸ‘«πŸ§‘πŸΏβ€πŸ€β€πŸ§‘πŸ»πŸ‘©πŸΎβ€πŸ€β€πŸ‘¨πŸΏπŸ‘¬πŸΏ > > https://github.com/hpcaitech/ColossalAI/blob/f313babd11f8137c2496e7dc54c6b61604cd3672/colossalai/zero/low_level/low_level_optim.py#L56 > > > Does colossai ai's zero2 support cpu offload? Yes, You...

The results of PPO training are not so good

Hi @guijuzhejiang There are many possible factors and parameters, and it's hard to give a best/good recommendation about them. We are also trying it with different models/datasets/etc. Welcome to share...

[BUG]:train_reward_model acc is low

Hi @guoweigang Maybe it's an inappropriate sh command. We have fixed it. Thanks. #3490 https://github.com/hpcaitech/ColossalAI/blob/main/applications/Chat/examples/train_rm.sh

Is the BLOOM model supported? How to perform task tuning like medical model with LLaMA33B？

Hi @delltower Yes, BLOOM and LLaMA have been supported. https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples#support-model Mainly by adding additional corresponding datasets

[BUG]: how can i fine-tuning the glm-130b based on colossal-ai

@lyzKF Yes, you can. Welcome to join the support of GLM https://github.com/hpcaitech/ColossalAI/pull/3567

[BUG]: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1, 1, 512, 128]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Hi @liulihui123 Could you please provide more details to help us to reproduce this issue? The information is now too limited to locate. Thanks.

[BUG]: CUDA out of memory. Tried to allocate 25.10 GiB

Hi @Tian14267 We have add How to train with limited resources. Thanks. https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat#faq

[FEATURE]: please add 8-bits NLP model training mode.

Hi @yynil We have add How to train with limited resources https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat#faq For 8bits training, we are not currently making it a high priority task due to the potential for...

build docker failed

Hi @densechen https://github.com/hpcaitech/ColossalAI#installation We do not currently support and test MacOS. Sorry about that.