ColossalAI

ColossalAI copied to clipboard

[FEATURE]: 2-stage DataParallel Load Sharded CheckPoint Strategy

1

### Describe the feature **Describe the feature:** `Training in a large scale env with multiple DP group strategy will load huge data load throughtput. Ex: Training LLAMA2-70B (the checkpoint is...

superleo

[BUG]: does colossalai only support full layers training?

3

### 🐛 Describe the bug i'm using the colossalai to train the animate anyone. and the training need to freeze some layers in model. but that would lead to assert...

zhangvia

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

FrankLeeeee

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

flybird11111

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [x] The title follows the standard format: `[doc/gemini/tensor/...]: A...

CjhHa1

### Discussed in https://github.com/hpcaitech/ColossalAI/discussions/5393 Originally posted by **zhpacer** February 20, 2024 CUDA: Build cuda_11.8.r11.8/compiler.31833905_0 following: https://github.com/hpcaitech/ColossalAI/tree/main/applications/Colossal-LLaMA-2 Error: utils/flash_attention_patch.py", line 22, in from colossalai.accelerator import get_accelerator ModuleNotFoundError: No module named 'colossalai.accelerator'

zhpacer

[ColossalChat] Update RLHF V2

3

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

YeAnbang

[BUG]:MetaTensor :RuntimeError: !check_has_torch_dispatch(obj)

3

### 🐛 Describe the bug return self._op(*args, **kwargs or {}) File "/opt/conda/lib/python3.9/site-packages/torch/_prims/__init__.py", line 292, in _backend_select_impl return meta(*args, **kwargs) File "/opt/conda/lib/python3.9/site-packages/torch/_prims/__init__.py", line 2410, in _iota_meta return torch.empty( File "/opt/conda/lib/python3.9/site-packages/colossalai/lazy/lazy_init.py", line...

hejianle

[BUG]: ImportError: cannot import name 'MOE_CONTEXT' from partially initialized module

2

### 🐛 Describe the bug 您好，当我只调用`from colossalai.nn.layer.moe import Experts, MoeLayer, Top1Router, NormalNoiseGenerator`时候出现以下问题： `Traceback (most recent call last): File "/home/songty/working_dir/moe/main.py", line 2, in from colossalai.nn.layer.moe import Experts, MoeLayer, Top1Router, NormalNoiseGenerator File...

songty21110133

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

digger-yu