Megatron-DeepSpeed icon indicating copy to clipboard operation
Megatron-DeepSpeed copied to clipboard

Published 20 hours ago • verified publisher icon bigscience-workshop

The difference between zero-3 and megatron with zero-2

Open nicosouth opened this issue 1 year ago • 0 comments

hi, I looked up a lot of information.

but I still don't understand the difference between zero-3 and megatron with zero-2.

they all split the model.

nicosouth avatar Aug 25 '23 04:08 nicosouth