YeAnbang

[ColossalChat][WIP] Refactor coati; Fix PPO, LORA; Add DPO

1

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

[ColossalChat] Update RLHF V2

3

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

### 🐛 Describe the bug I logged the loss when running the continue pretrain script. The loss is nan for some batches. It turns out these lines are buggy. https://github.com/hpcaitech/ColossalAI/blob/2dd01e3a1430f223b9ef8e61b73cf17f60fccb07/applications/Colossal-LLaMA-2/colossal_llama2/dataset/spliced_and_tokenized_dataset.py#L59C1-L62C55...

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

[PROPOSAL]: Does the LowLevelZero Plugin Support Lora, This Code Is Confusing

1

### Proposal Does the LowLevelZero Plugin Support Lora, This Code Is Confusing ### Self-service - [ ] I'd be willing to do some initial work on this proposal myself.

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...