Stas Bekman
Stas Bekman
Thank you, @Superjomn
@tjruwase, has the work started on this? Thank you!
That's great news, @xylian86 - there are quite a few folks hoping to speed up their large checkpoint conversion. So thank you for working on that! Your plan looks great...
unstale, this is still unresolved.
unstale, this is still unresolved.
unstale, this is still unresolved.
> I see that the AMP automatic mixed precision within the deepspeed config is not compatible with Zero, but is that a hard limitation? as in if i were to...
I have never used this option. I think it's a very old way of doing amp via the apex package. just enabling bf16 in deepspeed and no manual `amp` addition...
deepspeed's bf16 mode is very similar to torch amp's mixed precision - the master weights and grads are in fp32. All operation that need to accumulate in fp32, like torch's...
It definitely matters, accumulating bf16 data points in bf16 is quite lossy. It'd be a good idea to write a small repro to double check your measurement. I hope to...