About training phenomena in my experiment

[Dian-Yi]

I do some traing, one of train log: 2021-04-12 05:53:50,174 GPU name -> GeForce GTX 1070 2021-04-12 05:53:50,174 GPU feat -> _CudaDeviceProperties(name='GeForce GTX 1070', major=6, minor=1, total_memory=8117MB, multi_processor_count=15) 2021-04-12 05:53:50,174 {'CUDNN': CfgNode({'BENCHMARK': True, 'DETERMINISTIC': False, 'ENABLED': True}), 'DATASET': CfgNode({'SEQLEN': 16, 'OVERLAP': 0.5}), 'DEBUG': False, 'DEBUG_FREQ': 5, 'DEVICE': 'cuda', 'EXP_NAME': 'vibe', 'LOGDIR': 'results/vibe_tests/12-04-2021_05-53-49_vibe', 'LOSS': {'D_MOTION_LOSS_W': 0.5, 'KP_2D_W': 300.0, 'KP_3D_W': 300.0, 'POSE_W': 60.0, 'SHAPE_W': 0.06}, 'MODEL': {'TEMPORAL_TYPE': 'gru', 'TGRU': {'ADD_LINEAR': True, 'BIDIRECTIONAL': False, 'HIDDEN_SIZE': 1024, 'NUM_LAYERS': 2, 'RESIDUAL': True}}, 'NUM_WORKERS': 8, 'OUTPUT_DIR': 'results/vibe_tests', 'SEED_VALUE': -1, 'TRAIN': {'BATCH_SIZE': 32, 'DATASETS_2D': ['Insta'], 'DATASETS_3D': ['MPII3D'], 'DATASET_EVAL': 'ThreeDPW', 'DATA_2D_RATIO': 0.6, 'END_EPOCH': 30, 'GEN_LR': 5e-05, 'GEN_MOMENTUM': 0.9, 'GEN_OPTIM': 'Adam', 'GEN_WD': 0.0, 'LR_PATIENCE': 5, 'MOT_DISCR': {'ATT': {'DROPOUT': 0.2, 'LAYERS': 3, 'SIZE': 1024}, 'FEATURE_POOL': 'attention', 'HIDDEN_SIZE': 1024, 'LR': 0.0001, 'MOMENTUM': 0.9, 'NUM_LAYERS': 2, 'OPTIM': 'Adam', 'UPDATE_STEPS': 1, 'WD': 0.0001}, 'NUM_ITERS_PER_EPOCH': 500, 'PRETRAINED': '', 'PRETRAINED_REGRESSOR': 'data/vibe_data/spin_model_checkpoint.pth.tar', 'RESUME': '', 'START_EPOCH': 0}} 2021-04-12 05:56:12,234 => no checkpoint found at '' 2021-04-12 05:58:25,227 (500/500) | Total: 0:02:12 | ETA: 0:00:01 | loss: 2.6379 | loss_kp_2d: 1.21 | loss_kp_3d: 1.11 | e_m_disc_loss: 0.27 | d_m_disc_real: 0.18 | d_m_disc_fake: 0.08 | d_m_disc_loss: 0.27 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 05:58:29,604 (20/20) | batch: 36.53ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 05:58:36,495 Epoch 0, MPJPE: 96.8489, PA-MPJPE: 63.4485, ACCEL: 31.7180, PVE: 129.4680, ACCEL_ERR: 32.8076, 2021-04-12 05:58:36,511 Epoch 1 performance: 63.4485 2021-04-12 05:58:36,733 Best performance achived, saving it! 2021-04-12 06:00:50,967 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.4929 | loss_kp_2d: 0.75 | loss_kp_3d: 1.20 | e_m_disc_loss: 0.20 | d_m_disc_real: 0.09 | d_m_disc_fake: 0.09 | d_m_disc_loss: 0.18 | data: 0.00 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:00:55,266 (20/20) | batch: 36.32ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:01:01,011 Epoch 1, MPJPE: 91.8842, PA-MPJPE: 61.0828, ACCEL: 32.0496, PVE: 114.6013, ACCEL_ERR: 33.1420, 2021-04-12 06:01:01,026 Epoch 2 performance: 61.0828 2021-04-12 06:01:05,441 Best performance achived, saving it! 2021-04-12 06:03:22,396 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.4246 | loss_kp_2d: 0.93 | loss_kp_3d: 0.87 | e_m_disc_loss: 0.15 | d_m_disc_real: 0.12 | d_m_disc_fake: 0.12 | d_m_disc_loss: 0.24 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:03:26,727 (20/20) | batch: 36.35ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:03:33,831 Epoch 2, MPJPE: 95.6156, PA-MPJPE: 62.1054, ACCEL: 30.9210, PVE: 122.3789, ACCEL_ERR: 32.0565, 2021-04-12 06:03:33,847 Epoch 3 performance: 62.1054 2021-04-12 06:05:53,800 (500/500) | Total: 0:02:19 | ETA: 0:00:01 | loss: 2.2886 | loss_kp_2d: 0.93 | loss_kp_3d: 1.24 | e_m_disc_loss: 0.15 | d_m_disc_real: 0.16 | d_m_disc_fake: 0.13 | d_m_disc_loss: 0.28 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:05:58,208 (20/20) | batch: 36.74ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:06:04,375 Epoch 3, MPJPE: 98.2617, PA-MPJPE: 63.8534, ACCEL: 30.0421, PVE: 121.2747, ACCEL_ERR: 31.2873, 2021-04-12 06:06:04,391 Epoch 4 performance: 63.8534 2021-04-12 06:08:19,654 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.4323 | loss_kp_2d: 0.96 | loss_kp_3d: 1.23 | e_m_disc_loss: 0.34 | d_m_disc_real: 0.04 | d_m_disc_fake: 0.03 | d_m_disc_loss: 0.07 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:08:23,952 (20/20) | batch: 36.06ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:08:37,568 Epoch 4, MPJPE: 99.3865, PA-MPJPE: 62.9814, ACCEL: 29.7359, PVE: 128.0962, ACCEL_ERR: 30.9927, 2021-04-12 06:08:37,583 Epoch 5 performance: 62.9814 2021-04-12 06:10:52,786 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.4067 | loss_kp_2d: 0.94 | loss_kp_3d: 1.30 | e_m_disc_loss: 0.18 | d_m_disc_real: 0.08 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.18 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:10:57,175 (20/20) | batch: 36.18ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:11:03,432 Epoch 5, MPJPE: 96.4716, PA-MPJPE: 64.0070, ACCEL: 27.9572, PVE: 115.6693, ACCEL_ERR: 29.3275, 2021-04-12 06:11:03,447 Epoch 6 performance: 64.0070 2021-04-12 06:13:18,610 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.3227 | loss_kp_2d: 0.66 | loss_kp_3d: 0.88 | e_m_disc_loss: 0.13 | d_m_disc_real: 0.11 | d_m_disc_fake: 0.14 | d_m_disc_loss: 0.24 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:13:23,028 (20/20) | batch: 36.14ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:13:29,372 Epoch 6, MPJPE: 95.1898, PA-MPJPE: 61.6976, ACCEL: 28.9725, PVE: 118.4678, ACCEL_ERR: 30.2380, 2021-04-12 06:13:29,388 Epoch 7 performance: 61.6976 2021-04-12 06:15:44,500 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.2829 | loss_kp_2d: 0.74 | loss_kp_3d: 0.60 | e_m_disc_loss: 0.14 | d_m_disc_real: 0.12 | d_m_disc_fake: 0.12 | d_m_disc_loss: 0.24 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:15:49,108 (20/20) | batch: 36.52ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:15:55,332 Epoch 7, MPJPE: 102.5145, PA-MPJPE: 64.9837, ACCEL: 27.6890, PVE: 118.6595, ACCEL_ERR: 29.0881, 2021-04-12 06:15:55,348 Epoch 8 performance: 64.9837 2021-04-12 06:18:10,556 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.2667 | loss_kp_2d: 1.35 | loss_kp_3d: 1.23 | e_m_disc_loss: 0.16 | d_m_disc_real: 0.09 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.20 | data: 0.00 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:18:15,124 (20/20) | batch: 36.61ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:18:22,850 Epoch 8, MPJPE: 99.4365, PA-MPJPE: 64.0029, ACCEL: 27.7016, PVE: 117.9277, ACCEL_ERR: 29.0838, 2021-04-12 06:18:22,867 Epoch 9 performance: 64.0029 2021-04-12 06:20:39,679 (500/500) | Total: 0:02:16 | ETA: 0:00:01 | loss: 2.2280 | loss_kp_2d: 0.82 | loss_kp_3d: 0.66 | e_m_disc_loss: 0.16 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.22 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:20:44,497 (20/20) | batch: 36.73ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:20:50,461 Epoch 9, MPJPE: 101.0187, PA-MPJPE: 65.1288, ACCEL: 27.9159, PVE: 120.0184, ACCEL_ERR: 29.3211, 2021-04-12 06:20:50,477 Epoch 10 performance: 65.1288 2021-04-12 06:23:05,244 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1767 | loss_kp_2d: 0.91 | loss_kp_3d: 0.76 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.13 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.23 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:23:10,138 (20/20) | batch: 36.88ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:23:16,517 Epoch 10, MPJPE: 102.2744, PA-MPJPE: 64.9036, ACCEL: 27.5009, PVE: 119.9600, ACCEL_ERR: 28.9325, 2021-04-12 06:23:16,534 Epoch 11 performance: 64.9036 2021-04-12 06:25:31,958 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1841 | loss_kp_2d: 0.89 | loss_kp_3d: 1.04 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.21 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:25:37,141 (20/20) | batch: 39.76ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:25:43,976 Epoch 11, MPJPE: 101.6049, PA-MPJPE: 65.1903, ACCEL: 27.5021, PVE: 119.5197, ACCEL_ERR: 28.9424, 2021-04-12 06:25:44,002 Epoch 12 performance: 65.1903 2021-04-12 06:27:58,917 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.2065 | loss_kp_2d: 1.14 | loss_kp_3d: 1.30 | e_m_disc_loss: 0.16 | d_m_disc_real: 0.12 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.23 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:28:03,819 (20/20) | batch: 36.53ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:28:10,280 Epoch 12, MPJPE: 103.2477, PA-MPJPE: 66.3765, ACCEL: 27.3345, PVE: 121.7078, ACCEL_ERR: 28.7987, 2021-04-12 06:28:10,295 Epoch 13 performance: 66.3765 2021-04-12 06:30:28,694 (500/500) | Total: 0:02:16 | ETA: 0:00:01 | loss: 2.1426 | loss_kp_2d: 0.76 | loss_kp_3d: 0.91 | e_m_disc_loss: 0.18 | d_m_disc_real: 0.11 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.21 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:30:33,722 (20/20) | batch: 36.4ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:30:45,605 Epoch 13, MPJPE: 104.2243, PA-MPJPE: 66.9516, ACCEL: 28.2327, PVE: 121.4232, ACCEL_ERR: 29.6791, 2021-04-12 06:30:45,620 Epoch 14 performance: 66.9516 2021-04-12 06:33:00,353 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.2232 | loss_kp_2d: 0.79 | loss_kp_3d: 0.66 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.12 | d_m_disc_loss: 0.22 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:33:05,397 (20/20) | batch: 36.27ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:33:11,830 Epoch 14, MPJPE: 103.6547, PA-MPJPE: 66.7816, ACCEL: 28.3621, PVE: 121.3017, ACCEL_ERR: 29.7972, 2021-04-12 06:33:11,845 Epoch 15 performance: 66.7816 2021-04-12 06:35:26,669 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1401 | loss_kp_2d: 0.75 | loss_kp_3d: 1.50 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.12 | d_m_disc_loss: 0.21 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:35:31,773 (20/20) | batch: 36.57ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:35:38,926 Epoch 15, MPJPE: 103.5687, PA-MPJPE: 66.5759, ACCEL: 28.3627, PVE: 121.7134, ACCEL_ERR: 29.7988, 2021-04-12 06:35:38,941 Epoch 16 performance: 66.5759 2021-04-12 06:37:58,846 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1414 | loss_kp_2d: 0.70 | loss_kp_3d: 0.70 | e_m_disc_loss: 0.18 | d_m_disc_real: 0.11 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.22 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:38:04,096 (20/20) | batch: 36.33ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:38:11,175 Epoch 16, MPJPE: 102.7752, PA-MPJPE: 66.0851, ACCEL: 28.1922, PVE: 121.2658, ACCEL_ERR: 29.6264, 2021-04-12 06:38:11,191 Epoch 17 performance: 66.0851 2021-04-12 06:40:25,980 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1996 | loss_kp_2d: 0.88 | loss_kp_3d: 1.31 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.12 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.22 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:40:31,175 (20/20) | batch: 36.27ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:40:37,697 Epoch 17, MPJPE: 102.5017, PA-MPJPE: 65.8600, ACCEL: 28.0586, PVE: 121.2750, ACCEL_ERR: 29.4964, 2021-04-12 06:40:37,713 Epoch 18 performance: 65.8600 2021-04-12 06:42:54,856 (500/500) | Total: 0:02:16 | ETA: 0:00:01 | loss: 2.1536 | loss_kp_2d: 0.59 | loss_kp_3d: 0.59 | e_m_disc_loss: 0.15 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.13 | d_m_disc_loss: 0.23 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:43:00,163 (20/20) | batch: 36.41ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:43:05,932 Epoch 18, MPJPE: 102.3520, PA-MPJPE: 65.7480, ACCEL: 27.9462, PVE: 121.2409, ACCEL_ERR: 29.3880, 2021-04-12 06:43:05,948 Epoch 19 performance: 65.7480 2021-04-12 06:45:22,061 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1832 | loss_kp_2d: 0.81 | loss_kp_3d: 1.51 | e_m_disc_loss: 0.14 | d_m_disc_real: 0.11 | d_m_disc_fake: 0.14 | d_m_disc_loss: 0.24 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:45:27,391 (20/20) | batch: 36.56ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:45:33,061 Epoch 19, MPJPE: 102.0669, PA-MPJPE: 65.7208, ACCEL: 27.9299, PVE: 121.0225, ACCEL_ERR: 29.3638, 2021-04-12 06:45:33,077 Epoch 20 performance: 65.7208 2021-04-12 06:47:51,773 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1986 | loss_kp_2d: 0.65 | loss_kp_3d: 0.99 | e_m_disc_loss: 0.18 | d_m_disc_real: 0.11 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.21 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:47:57,080 (20/20) | batch: 36.3ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:48:02,806 Epoch 20, MPJPE: 102.1162, PA-MPJPE: 65.7059, ACCEL: 27.9246, PVE: 121.0233, ACCEL_ERR: 29.3590, 2021-04-12 06:48:02,821 Epoch 21 performance: 65.7059 2021-04-12 06:50:17,633 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1834 | loss_kp_2d: 1.75 | loss_kp_3d: 0.79 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.11 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.22 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:50:22,973 (20/20) | batch: 36.3ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:50:28,866 Epoch 21, MPJPE: 102.2383, PA-MPJPE: 65.7585, ACCEL: 27.9185, PVE: 121.0941, ACCEL_ERR: 29.3542, 2021-04-12 06:50:28,881 Epoch 22 performance: 65.7585 2021-04-12 06:52:43,673 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1721 | loss_kp_2d: 0.69 | loss_kp_3d: 0.64 | e_m_disc_loss: 0.17 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.21 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:52:49,031 (20/20) | batch: 36.31ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:52:55,032 Epoch 22, MPJPE: 102.2184, PA-MPJPE: 65.7504, ACCEL: 27.9173, PVE: 121.0417, ACCEL_ERR: 29.3533, 2021-04-12 06:52:55,047 Epoch 23 performance: 65.7504 2021-04-12 06:55:14,479 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1915 | loss_kp_2d: 1.02 | loss_kp_3d: 0.65 | e_m_disc_loss: 0.19 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.20 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:55:19,852 (20/20) | batch: 36.19ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:55:26,227 Epoch 23, MPJPE: 102.2664, PA-MPJPE: 65.7846, ACCEL: 27.9176, PVE: 121.0718, ACCEL_ERR: 29.3540, 2021-04-12 06:55:26,243 Epoch 24 performance: 65.7846 2021-04-12 06:57:42,693 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1074 | loss_kp_2d: 0.98 | loss_kp_3d: 0.57 | e_m_disc_loss: 0.16 | d_m_disc_real: 0.14 | d_m_disc_fake: 0.12 | d_m_disc_loss: 0.26 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 06:57:48,100 (20/20) | batch: 36.0ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 06:57:54,911 Epoch 24, MPJPE: 102.2860, PA-MPJPE: 65.7996, ACCEL: 27.9114, PVE: 121.0974, ACCEL_ERR: 29.3485, 2021-04-12 06:57:54,927 Epoch 25 performance: 65.7996 2021-04-12 07:00:14,144 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1095 | loss_kp_2d: 0.77 | loss_kp_3d: 1.37 | e_m_disc_loss: 0.18 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.20 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 07:00:19,496 (20/20) | batch: 36.24ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 07:00:25,485 Epoch 25, MPJPE: 102.3316, PA-MPJPE: 65.8409, ACCEL: 27.9090, PVE: 121.1213, ACCEL_ERR: 29.3470, 2021-04-12 07:00:25,500 Epoch 26 performance: 65.8409 2021-04-12 07:02:43,904 (500/500) | Total: 0:02:15 | ETA: 0:00:01 | loss: 2.1383 | loss_kp_2d: 0.98 | loss_kp_3d: 0.75 | e_m_disc_loss: 0.18 | d_m_disc_real: 0.12 | d_m_disc_fake: 0.11 | d_m_disc_loss: 0.22 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 07:02:49,251 (20/20) | batch: 36.15ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 07:02:58,195 Epoch 26, MPJPE: 102.3399, PA-MPJPE: 65.8435, ACCEL: 27.9090, PVE: 121.1264, ACCEL_ERR: 29.3471, 2021-04-12 07:02:58,210 Epoch 27 performance: 65.8435 2021-04-12 07:05:18,338 (500/500) | Total: 0:02:19 | ETA: 0:00:01 | loss: 2.1889 | loss_kp_2d: 1.31 | loss_kp_3d: 0.53 | e_m_disc_loss: 0.15 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.13 | d_m_disc_loss: 0.23 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 07:05:23,886 (20/20) | batch: 37.91ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 07:05:29,638 Epoch 27, MPJPE: 102.3496, PA-MPJPE: 65.8491, ACCEL: 27.9090, PVE: 121.1336, ACCEL_ERR: 29.3472, 2021-04-12 07:05:29,656 Epoch 28 performance: 65.8491 2021-04-12 07:07:44,528 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1811 | loss_kp_2d: 1.10 | loss_kp_3d: 1.17 | e_m_disc_loss: 0.19 | d_m_disc_real: 0.10 | d_m_disc_fake: 0.09 | d_m_disc_loss: 0.19 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 07:07:49,968 (20/20) | batch: 36.08ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 07:07:55,670 Epoch 28, MPJPE: 102.3576, PA-MPJPE: 65.8521, ACCEL: 27.9087, PVE: 121.1393, ACCEL_ERR: 29.3470, 2021-04-12 07:07:55,686 Epoch 29 performance: 65.8521 2021-04-12 07:10:14,340 (500/500) | Total: 0:02:14 | ETA: 0:00:01 | loss: 2.1747 | loss_kp_2d: 0.60 | loss_kp_3d: 0.55 | e_m_disc_loss: 0.19 | d_m_disc_real: 0.09 | d_m_disc_fake: 0.10 | d_m_disc_loss: 0.19 | data: 0.01 | forward: 0.06 | loss: 0.00 | backward: 0.20 | batch: 0.27 2021-04-12 07:10:19,843 (20/20) | batch: 36.58ms | Total: 0:00:03 | ETA: 0:00:01 2021-04-12 07:10:25,561 Epoch 29, MPJPE: 102.3593, PA-MPJPE: 65.8523, ACCEL: 27.9086, PVE: 121.1381, ACCEL_ERR: 29.3470, 2021-04-12 07:10:25,576 Epoch 30 performance: 65.8523

Here is my training summary:

1. train data: INSTA MPII3D, three times traing PA-MJPE test results(get from eval.py ):56.8, 58.6, 57.8。 you can find their difference is a little big. Is this normal？
2. If you dont use PRETRAINED_REGRESSOR in config, you maybe meet error: " 'MPJPE error is {performance}, higher than 80.0. Exiting!...'" . I think the pretrained model 'spin_model_checkpoint.pth.tar' is very import, but i cant find how it is generated. In paper, this part is belong to regressor.
3. I trained all data: Insta, PoseTrack, PennAction, 3DPW, MPII3D. but it's MPJPE not be better. Are some datasets not critical in this work?

thanks for someone reply！！！