ColossalAI

ColossalAI copied to clipboard

CjhHa1

## 📌 Checklist before creating the PR - [x] I have created an issue for this PR for traceability - [x] The title follows the standard format: `[doc/gemini/tensor/...]: A concise...

isky-cd

[lora] support lora for Gemini

3

## 📌 Checklist before creating the PR - [x] I have created an issue for this PR for traceability - [x] The title follows the standard format: `[doc/gemini/tensor/...]: A concise...

Fridge003

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

KKZ20

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

ver217

## 📌 Checklist before creating the PR - [x] I have created an issue for this PR for traceability - [x] The title follows the standard format: `[doc/gemini/tensor/...]: A concise...

Camille7777

[BUG]: Colossalai-OpenMoE-8b : loss value is very large and cannot converge

4

### 🐛 Describe the bug I am currently running the Colossalai/examples/language/openmoe project with the following experimental setup: datasets: load_dataset("yizhongw/self_instruct/data/finetuning/self_instruct_221203", "super_natural_instructions"), model: openmoe-8b A100_num_GPU：8 Epoch：3 BatchSize：4 LR：0.00001 zero_stage：1 precision：bf16 Boost plugin：ep_zero...

hangchen426926

…se.py code style ## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the...

NonvolatileMemory

[shardformer] fix pipeline forward error if custom layer distribution is used

3

…distribution ## 📌 Checklist before creating the PR - [x] I have created an issue for this PR for traceability - [x] The title follows the standard format: `[doc/gemini/tensor/...]: A...

insujang

[BUG]: Colossalai-OpenMoE example failed to converge

2

### 🐛 Describe the bug # Setup I am currently running the Colossalai/examples/language/openmoe project with the following experimental setup: - datasets: `load_dataset("yizhongw/self_instruct", "super_natural_instructions")`, I also tried `"wikitext-2"` - model: `openmoe-base`...

marsggbo