tfoptflow icon indicating copy to clipboard operation
tfoptflow copied to clipboard

Published 20 hours ago • verified publisher icon philferriere

Loss didn't decrease in pwcnet_train_sm-6-2-multisteps-chairsthingsmix

Open xuleyoung opened this issue 4 years ago • 1 comments

some output infos: 2019-10-31 21:27:13 Iter 49000 [Train]: loss=184.37, epe=15.17, lr=0.000100, samples/sec=24.4, sec/step=0.655, eta=8 days, 17:27:22 2019-10-31 21:27:21 Iter 49000 [Val]: loss=141.03, epe=11.58 Saving model... INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-49000 is not in all_model_checkpoint_paths. Manually adding it. ... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-49000 2019-10-31 21:39:36 Iter 50000 [Train]: loss=184.73, epe=15.20, lr=0.000100, samples/sec=27.7, sec/step=0.578, eta=7 days, 16:41:55 2019-10-31 21:39:44 Iter 50000 [Val]: loss=140.98, epe=11.57 Saving model... INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-50000 is not in all_model_checkpoint_paths. Manually adding it. ... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-50000 2019-10-31 21:50:47 Iter 51000 [Train]: loss=184.02, epe=15.14, lr=0.000100, samples/sec=28.3, sec/step=0.566, eta=7 days, 12:37:08 2019-10-31 21:50:56 Iter 51000 [Val]: loss=140.30, epe=11.52 Saving model... INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-51000 is not in all_model_checkpoint_paths. Manually adding it. ... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-51000 2019-10-31 22:03:22 Iter 52000 [Train]: loss=184.69, epe=15.20, lr=0.000100, samples/sec=26.0, sec/step=0.616, eta=8 days, 4:35:21 2019-10-31 22:03:33 Iter 52000 [Val]: loss=141.20, epe=11.60 Saving model... INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-52000 is not in all_model_checkpoint_paths. Manually adding it. ... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-52000 2019-10-31 22:16:15 Iter 53000 [Train]: loss=184.40, epe=15.17, lr=0.000100, samples/sec=25.4, sec/step=0.629, eta=8 days, 8:18:36 2019-10-31 22:16:25 Iter 53000 [Val]: loss=140.56, epe=11.53 Saving model... INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-53000 is not in all_model_checkpoint_paths. Manually adding it. ... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-53000

xuleyoung avatar Oct 31 '19 14:10 xuleyoung

I change the lr, however, there is no obvious loss decrese.

xuleyoung avatar Nov 02 '19 08:11 xuleyoung