Palette-Image-to-Image-Diffusion-Models
Palette-Image-to-Image-Diffusion-Models copied to clipboard
Janspiry
Training loss growing up
Does anyone has problem like this? I just use the colorization config for my data, and the loss growing up and the output always be noisy.
23-09-23 23:32:14.272 - INFO: Begin model train. 23-09-23 23:39:51.920 - INFO: train/mse_loss: 0.5875651128242876 23-09-23 23:39:51.921 - INFO: epoch: 1 23-09-23 23:39:51.921 - INFO: iters: 6008 23-09-23 23:44:58.423 - INFO: train/mse_loss: 0.18443983706285458 23-09-23 23:47:28.679 - INFO: train/mse_loss: 0.1707256488630997 23-09-23 23:47:28.679 - INFO: epoch: 2 23-09-23 23:47:28.679 - INFO: iters: 12016 23-09-23 23:55:01.374 - INFO: train/mse_loss: 0.12716468866324218 23-09-23 23:55:01.375 - INFO: epoch: 3 23-09-23 23:55:01.375 - INFO: iters: 18024 23-09-23 23:57:34.015 - INFO: train/mse_loss: 0.11658619797452982 23-09-24 00:02:33.583 - INFO: train/mse_loss: 0.12064346502449477 23-09-24 00:02:33.584 - INFO: epoch: 4 23-09-24 00:02:33.584 - INFO: iters: 24032 23-09-24 00:10:04.679 - INFO: train/mse_loss: 0.1233654677937801 23-09-24 00:10:08.177 - INFO: train/mse_loss: 0.12338643875741927 23-09-24 00:10:08.178 - INFO: epoch: 5 23-09-24 00:10:08.178 - INFO: iters: 30040 23-09-24 00:10:08.178 - INFO:
------------------------------Validation Start------------------------------ 23-09-24 00:17:50.267 - INFO: val/mae: 1.152842402458191 23-09-24 00:17:50.267 - INFO: ------------------------------Validation End------------------------------
23-09-24 00:25:21.371 - INFO: train/mse_loss: 0.1386190212509048 23-09-24 00:25:21.372 - INFO: epoch: 6 23-09-24 00:25:21.372 - INFO: iters: 36058 23-09-24 00:32:51.582 - INFO: train/mse_loss: 0.16339113016162668 23-09-24 00:32:51.584 - INFO: epoch: 7 23-09-24 00:32:51.584 - INFO: iters: 42066 23-09-24 00:40:20.440 - INFO: train/mse_loss: 0.20504292834813045 23-09-24 00:40:20.441 - INFO: epoch: 8 23-09-24 00:40:20.441 - INFO: iters: 48074 23-09-24 00:47:50.634 - INFO: train/mse_loss: 0.26625456471497144 23-09-24 00:47:50.635 - INFO: epoch: 9 23-09-24 00:47:50.635 - INFO: iters: 54082 23-09-24 00:55:21.751 - INFO: train/mse_loss: 0.3486290028425412 23-09-24 00:55:21.752 - INFO: epoch: 10 23-09-24 00:55:21.752 - INFO: iters: 60090 23-09-24 00:55:21.752 - INFO: Saving the self at the end of epoch 10 23-09-24 00:55:40.332 - INFO:
------------------------------Validation Start------------------------------ 23-09-24 01:03:16.369 - INFO: val/mae: 1.0535274744033813 23-09-24 01:03:16.370 - INFO: ------------------------------Validation End------------------------------
23-09-24 01:10:41.844 - INFO: train/mse_loss: 0.4689194846169133 23-09-24 01:10:41.845 - INFO: epoch: 11 23-09-24 01:10:41.845 - INFO: iters: 66108 23-09-24 01:15:32.251 - INFO: train/mse_loss: 0.587346616041256 23-09-24 01:18:05.937 - INFO: train/mse_loss: 0.6161996504755693 23-09-24 01:18:05.937 - INFO: epoch: 12 23-09-24 01:18:05.937 - INFO: iters: 72116 23-09-24 01:25:27.781 - INFO: train/mse_loss: 0.7735867933252363 23-09-24 01:25:27.782 - INFO: epoch: 13 23-09-24 01:25:27.782 - INFO: iters: 78124 23-09-24 01:27:54.342 - INFO: train/mse_loss: 0.8663824072270505 23-09-24 01:32:54.690 - INFO: train/mse_loss: 0.9051068551372117 23-09-24 01:32:54.690 - INFO: epoch: 14 23-09-24 01:32:54.690 - INFO: iters: 84132 23-09-24 01:40:07.631 - INFO: train/mse_loss: 0.9838901592422361 23-09-24 01:40:18.302 - INFO: train/mse_loss: 0.9842592114694267 23-09-24 01:40:18.302 - INFO: epoch: 15 23-09-24 01:40:18.302 - INFO: iters: 90140 23-09-24 01:40:18.302 - INFO:
------------------------------Validation Start------------------------------ 23-09-24 01:47:51.676 - INFO: val/mae: 1.1024448871612549 23-09-24 01:47:51.677 - INFO:
