opacus icon indicating copy to clipboard operation
opacus copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

fp16 support

Open ffuuugor opened this issue 2 years ago • 2 comments

(possibly with autocast and loss scaling, but in our experience with Alex this may result in training instabilities). On the other hand, it reduces memory usage and increases speed (roughly a factor 2 for both).

ffuuugor avatar Mar 11 '22 16:03 ffuuugor

Hi, I wonder whether opacus support fp16. In my experiment, I use the fp16 but get the error:

File "/home/yxguo/anaconda3/envs/ditto/lib/python3.7/site-packages/apex/amp/handle.py", line 110, in scale_loss if not optimizer._amp_stash.params_have_scaled_gradients: AttributeError: 'DPOptimizer' object has no attribute '_amp_stash'

yuxiang-guo avatar Jun 22 '22 05:06 yuxiang-guo

Hi As of this moment opacus doesn't support fp16.

This issue is created to track progress of adding the support. As of today it's not something planned for the near future (but we always welcome external pull requests)

ffuuugor avatar Jun 24 '22 15:06 ffuuugor