nanoGPT icon indicating copy to clipboard operation
nanoGPT copied to clipboard

Published 20 hours ago • verified publisher icon karpathy

Using float16 via Gradscaler

Open acheong08 opened this issue 2 years ago • 1 comments

https://huggingface.co/facebook/opt-30b

Larger open source model? Does it work?

acheong08 avatar Jan 09 '23 05:01 acheong08

It seems OPT uses float16. https://github.com/karpathy/nanoGPT/blob/3e0fd425794e3e0160ae6132916beff78823888a/train.py#L85-L88

How do I use gradscaler?

acheong08 avatar Jan 13 '23 01:01 acheong08