lightweight-gan
lightweight-gan copied to clipboard
lucidrains
how does this compare to stylegan2-ada?
A couple things I've noticed when comparing the two
- SG2-ADA has non-destructive augmentation via 'ADA' which augments without eventually adding holes / edges / funny colors to the generator
- I believe SG2-ADA has more built-in augmentation methods including more geometric transformations and color types
- SG2-ADA appears to have more robust FP16 support on Ampere cards - so I can train much larger models more easily on it than even the lightweight SG2 here. I believe this is a fairly easy enhancement on Lucid's great code though (easier said by me than done though!)
- Lucid's has better gradient accumulation support (I can't figure it out in ADA)
- Lucid's has attention layers, which I'm not sure that SG2 has
ADA is a strategy for varying the augmentation probability during the training, while the probability is constant in this repository.
Moreover, as mentioned above, different augmentations are offered in both repositories, although there is an overlap, e.g. color.
I wonder if adding more augmentation types is a high value addition for the Lucid implementation - reason being that I get "mind blowingly good" results, and quickly, from SG2-ADA (usually with just 5-10% augmentation within the first 48h of training) while the Lucid one takes longer and tends to give me more destructive results (model learns to put square cutouts into my generated images, or I get black borders from the translation augment etc). It may not require adaptive augmentation because we're talking about 5-10% only ... just more variety of augmentation to improve generalization without memorizing cutout in 10% of the generator outputs.