stylegan2-pytorch
stylegan2-pytorch copied to clipboard
--fp16 Slower & Does Not Reduce Memory Use
Hey there @lucidrains,
Came across your incredible work and immediately tried it out on my RTX 2070! Since the training will take some time and require a lot of memory, I was relieved that we can use APEX/Amp to train the model by simply adding the --fp16
option.
Unfortunately for me, the memory usage does not reduce compared to the regular fp32 training and the training speed was slower too.
Came across a similar issue #129 but it was closed before a fix was checked in. Will you still continue to work on fp16? I believe this will help many of your users (and fans!)
Hi Raye! I don't know why, but mixed precision no longer brings the memory down. It's the same when I tried to switch to amp. I'm not sure what's wrong, but I'm out of time (moving). Maybe someone else can figure this out!
@RayeRTX are you getting good results? please share :)
A bit unrelated, but I can't even get it running - I just keep getting NaN errors and the learning shutdowns.
@tannisroot yea, I get that feedback a lot. I think I will just remove this feature from the readme and keep it as a silent feature. Perhaps someone can help figure out what's wrong. It has worked for me in the past, so I'm not sure what changed
Still trying out various settings, lets see what we get!
Any chance someone figured out why fp16 is not working?
@lucidrains added Pytorch's Amp to his Lightweight-GAN repo and it works great on my Titan RTX!
@TKassis Thats very useful info! Do you find lightweight gan to work better for you?
@TKassis Thats very useful info! Do you find lightweight gan to work better for you?
It trains much much faster, but I haven't compared the two on the same training data.