Hongxin Liu

Did you try torchrun?

As it just shows training process, we just use randomly initialized model for simplicity. You can use a pretrained model easily by `GPTActor(pretrained='gpt2')`.

Hi, these environment variables should be set carefully. We generally recommend that users can launch by [torchrun](https://pytorch.org/docs/stable/elastic/run.html?highlight=torchrun)

Hi, old version of ZeRO will be be deprecated in the future. According to our benchmark results, new version of ZeRO is better than the old one. Could you tell...

> I think it might be hard to satisfy torch.all_close condition. checking the norm of weights might be an alternative. We will fix the random seed before each OP. All...

Torch's version in CI is 1.11, which is incompatible with meta tensor. I run test on local machine: 

Thanks for your suggestion. We will make efforts to implement this feature in this month.

Could you add unit tests to check this feature?

> > Could you add unit tests to check this feature? > > I've tested in my local experiments  > > But how to test it in unittest: colossalai/tests...

Thanks! We will take a look.