tutel icon indicating copy to clipboard operation
tutel copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Tutel with pytorch automatic mixed precision package

Open MiZhenxing opened this issue 2 years ago • 2 comments

Hi, thank you for your excellent package. I wonder whether tutel could be used seamless together with the automatic mixed precision package of PyTorch. If so, could you provide some hints on how to use it or provide an example python script? Thank you very much!

MiZhenxing avatar May 10 '22 13:05 MiZhenxing

Yes, Tutel is able to support native Pytorch AMP. Please follow this example: https://github.com/microsoft/tutel/blob/main/tutel/examples/helloworld_amp.py#L76

Note that you need to use @autocast() and with autocast: properly according to pytorch's doc. Welcome to report problems if you encounter any issue.

ghostplant avatar May 10 '22 17:05 ghostplant

Thanks for your quick response. Sorry I didn't notice that example 🤣. I will do some testing on my code.

MiZhenxing avatar May 11 '22 06:05 MiZhenxing