fairseq icon indicating copy to clipboard operation
fairseq copied to clipboard

Published 20 hours ago • verified publisher icon facebookresearch

bf16 with A100 GPUs

Open calderma opened this issue 2 years ago • 2 comments

A100 GPUs should support bf16 unless I am mistaken. I see it's currently only supported with TPUs. Are there any plans to support this?

calderma avatar Sep 05 '22 00:09 calderma

Any progress on this ? My initial thought was to just pass the appropriate data type in the call to torch.cuda.amp.autocast in the below fairseq_task.py file. image

Not sure if it would required changes anywhere else. I presume underlying PyTorch should handle the rest of it. Any pointers on that ?

harishankar-gopalan avatar Apr 30 '23 15:04 harishankar-gopalan