Jiarui Fang（方佳瑞） issues

Results 63 issues of


                                            Jiarui Fang（方佳瑞）

[Gemini] more rigorous unit tests for run_fwd_bwd

### What's the PR for Previously, Gemini is only checked on GPT2. After increasing the types of test models, the unitest test fails. You can merge this PR. But @1SAA...

Run Build and Test

[Do not merge] identify a bug for runtime tracer

bert grad_checkpoint = True时，runtime tracer和gemini的tracer检测的采样点数对不上。

[version] 0.1.11rc5 -> 0.1.12

Run Build and Test

[Gemini] chunk init use OrderedParamGenerator

Run Build and Test

[example] simplify opt example

### Why Make sure the user can run OPT to profile performance in 1 minute. No data download, no complex training parameter setting. Just simply run a few iterations.

[DO NOT MERGE] evaluation works for palm train.py

I convert this to draft, because GeminiDDP can not run inference. The states will be wrong.

[zero] low level optim supports ProcessGroup

## What's new This PR removes the dependency of LowLevelZeroOptimizer on gpc. gpc is a global variable. If use it, we can not use LowLevelZeroOptimizer together with ColoTensor TP. This...

Run Build and Test

[CI] add test_ci.sh for palm, opt and gpt

[FEATURE]: Unify interface of Gemini(offload) + ZeRO1 + ZeRO2 + ZeRO3

### Describe the feature ## Current States Currently, GeminiDDP has to shard the optimizer state, and it has covered zero3 and zero3+offload. However, It doesn't actually cover zero1 and zero3....

enhancement