Hao Ge

[BUG]: run gpt sample error: torch.distributed.elastic.multiprocessing.errors.ChildFailedError

5

### 🐛 Describe the bug I use https://github.com/hpcaitech/ColossalAI-Examples to start GPT2 training example. But it seems run train_gpt.py FAILED. Can anyone give some helps? the code root: `ColossalAI-Examples/language/gpt` my script:...

I found that you compare the performance for udp based on simplebaseline, can you release that part code and weights? thanks!

[BUG]DeepSpeed zero3 use more gpu memory than zero2

3

**Describe the bug** 1. DeepSpeed zero3 use more gpu memory than zero2. 2. Ulysess performance problem. **To Reproduce** Steps to reproduce the behavior: just use ulysess sample code: https://github.com/microsoft/Megatron-DeepSpeed/blob/main/examples_deepspeed/sequence_parallel/ds_pretrain_gpt_1.3B_seq_parallel_32k.sh change...

Why requires_grad attribute of weight from offloading will set to False ?

1

https://github.com/NVIDIA/TransformerEngine/blob/e3bb24e5a347c58353e62307bc84cca856f9e9be/transformer_engine/pytorch/module/linear.py#L405-L407 if the weight.requires_grad set to False, when to calculate and accumulate weight grads?