ColossalAI
ColossalAI copied to clipboard
hpcaitech
可以支持mistral模型吗
Describe the feature
You are using a model of type mistral to instantiate a model of type llama. This is not supported for all configurations of models and can yield errors
Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑🤝🧑👫🧑🏿🤝🧑🏻👩🏾🤝👨🏿👬🏿
Title: Can the mistral model be supported?
hi，mistral模型已经支持了，可以关注一下这个pr：https://github.com/hpcaitech/ColossalAI/pull/5103
hi，mistral模型已经支持了，可以关注一下这个pr：#5103
我的代码如下：
torchrun --standalone --nproc_per_node=8 train_sft.py
--pretrain ./zephyr-7b-beta
--tokenizer ./zephyr-7b-beta
--model 'llama'
--strategy colossalai_zero2_cpu
--save_path zephyr-7b-beta-sft
--dataset train_sft.json
--batch_size 1
--accumulation_steps 8
--lr 2e-5
--max_len 4096
--max_epochs 3
--grad_checkpoint
You are using a model of type mistral to instantiate a model of type llama. This is not supported for all configurations of models and can yield errors
@flybird11111
抱歉，shardformer已经支持了mistral，coati还未支持，我跟相关同事反馈一下。
shardformer何时支持下QWEN？
Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑🤝🧑👫🧑🏿🤝🧑🏻👩🏾🤝👨🏿👬🏿
@flybird11111
Sorry, shardformer already supports mistral, but coati does not yet support it. I will give feedback to relevant colleagues.
When will shardformer support QWEN?
@flybird11111
抱歉，shardformer已经支持了mistral，coati还未支持，我跟相关同事反馈一下。
shardformer何时支持下QWEN？
hi，qianwen会尽快支持的。chat也会尽快支持mistral