Add hypernetwork weight decay in training

[matthewdm0816]

Add an option to add weight decay (l2 regularization) in training. It seems the hypernetwork maybe overfit over training procedure, adding simple weight decay might help.

[HunterVacui]

FWIW I've been having difficulty training my own embeddings (not hypernetworks) and I've been thinking that some decay options could be useful there too. Start training with broad strokes and then decrease sensitivity over time for fine-tuning

[AUTOMATIC1111]

there already is a way to specify lr schedule: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Textual-Inversion#training-an-embedding

Does this clash with that? Is this even needed if you can use that?

[matthewdm0816]

Need more experiments. I tried much lower LR (say, 2e-7) after some steps, and the problem continues on making noisy image output. But lack of evidence of solving the problem with weight decay. The original idea comes from novel who states they used strong penalty on hypernetwork to prevent overfitting.