Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

[cdj0311]

trafficstars

hi, I have trained a GPT model with 4 nodes (8 gpus per node), but when I load the checkpoint with 6 nodes, I get the follows error:

deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 4 but the current world size is 6. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

[chenzhanyiczy]

hi, I have trained a GPT model with 4 nodes (8 gpus per node), but when I load the checkpoint with 6 nodes, I get the follows error:

deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 4 but the current world size is 6. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

hi, can you share your experience? I also encountered the same problem

[sasaadi]

Hi, Can you share your solution to this? thanks

[zhao0306]

Hi, can you share your solution to this problem? thanks. @cdj0311

[JinmingZhao]

Same problem.

[liutaocode]

Same Issue

[tommy-qichang]

Same here.

[cmstudyscode]

same here +1

[Rafa-zy]

same here + 1

[xjwang4]

same here + 1

[gugarosa]

Same here. Is there any guidance on how to proceed?

[young-chao]

Can you share your solution to me? thanks. @cdj0311

[macabdul9]

Any update on this? @cdj0311

[kxgong]

The problem is caused by the ckpt is saved on 6 gpus but is reloaded with 4 gpus.

Try torch.load(..., map_location=[torch.device('cpu'))

[young-chao]

你好，你的邮件已收到，谢谢！

[ybdesire]

same issue + 1

[timpal0l]

Same here;

deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 7 but the current world size is 8. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

[samadejacobs]

We recommend that you use DeepSpeed universal checkpoint.