kohya-ss
train_network.py: add debugging print for tensor and gradient norms
This is a small, but extremely helpful debug option for printing the gradient and model norms during training. I use it for debugging LR related issues.
Rather than doing print statements, would this be better as being a Tensorflow graph output? Sd-scripts already supports outputs like the LoRA key length as graphs in Tensorflow, so adding your output that way would be consistent with what's there already, and more visual.
It also wouldn't need an additional command line parameter that way, cause you can pick and choose what you want to see in Tensorflow, so the trace could always just be generated if it's useful.
Rather than doing print statements, would this be better as being a Tensorflow graph output? Sd-scripts already supports outputs like the LoRA key length as graphs in Tensorflow, so adding your output that way would be consistent with what's there already, and more visual.
It also wouldn't need an additional command line parameter that way, cause you can pick and choose what you want to see in Tensorflow, so the trace could always just be generated if it's useful.
I don't use Tensorboard, so I don't know how to add it there. But if you can send a commit, why not log to both?
This calculation is not free, so I don't want to do it every iteration. The frequency is mostly due to that, not to reduce the output volume.
It took me a while to find Tensorboard (which is the analysis part of Tensorflow) when using sd-scripts, because I didn't realize how easy it is to actually get working.
If you want to try it so you can see what I mean, it's as simple as:
pip install tensorboard
and then make sure you pass --logging_dir="/home/wherever/kohya/log" as a command line parameter into sd-scripts when running it.
And then from a command line window, start the Tensorboard UI with:
tensorboard --logdir=/home/wherever/kohya/log
and then follow the web link (it's a local address on your own machine), and you'll see the graphs I mean. They can be very helpful in understanding what's going on when training.