[WIP] Update hybrid model parallel GPT2 example for up to 175B parameters

[kurisusnowdeng]

• Added scripts for ColossalAI, Megatron-Deepspeed and Pytorch FSDP

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4032076207. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4041761263. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4050296375. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4053221865. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4060527002. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4063353025. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4070831489. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4071003095. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4071035062. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4072962674. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4073325210. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4073477316. Read our "CONTRIBUTING.md" for more reference to the code style.

[github-actions[bot]]

Your pre-commit check failed, follow the steps to run pre-commit on your file for code style consistency.

1. install pre-commit via "pip install pre-commit"
2. install pre-commit hooks via "pre-commit install"
3. run pre-commit on file with format error via "pre-commit run --files path" by replacing "path" with the actual file path
4. commit and push to your branch

View your job at https://github.com/hpcaitech/ColossalAI/actions/runs/4082792027. Read our "CONTRIBUTING.md" for more reference to the code style.

[FrankLeeeee]

@kurisusnowdeng is this PR still valid? If not, I will close it.

[kurisusnowdeng]

@kurisusnowdeng is this PR still valid? If not, I will close it.

@FrankLeeeee We're still working on this PR. Basically some evaluations are left, including gemini & 8bit optimizer pretraining performance, as well as unitests.

[FrankLeeeee]

Ok, I will convert this PR to draft for now as commits will trigger the CI repeatedly. @kurisusnowdeng do make it ready for review once it is ready :)

[kurisusnowdeng]

Ok, I will convert this PR to draft for now as commits will trigger the CI repeatedly. @kurisusnowdeng do make it ready for review once it is ready :)

@FrankLeeeee Thanks a lot