Hi, It has a problem of "E2E_RESULTS: Precision: 0, recall: 0.0, hmean: 0" after training.

[yub2nn]

Hello! We conducted training with the code below using the totaltext dataset. But score is like this "E2E_RESULTS: precision: 0, recall: 0.0, hmean: 0" after training. And as a result of visualization, neither bbox nor text were observed in the image.

$ python projects/SWINTS/train_net.py --num-gpus 1 --config-file projects/SWINTS/configs/SWINTS-swin-pretrain.yaml

Due to CUDA out of memory in the process, the parameter was modified as follows.

Base-SWINTS_swin.yaml

DATALOADER: NUM_WORKERS: 1 SOLVER: IMS_PER_BATCH: 1

SWINTS-swin-pretrain.yaml

SOLVER: STEPS: (360000,420000) MAX_ITER: 1000 CHECKPOINT_PERIOD: 1000

The log is shown below. I'll be waiting for the reply. Thank you!

[08/23 11:41:05 d2.engine.train_loop]: Starting training from iteration 0 [08/23 11:41:17 d2.utils.events]: eta: 0:08:23 iter: 19 total_loss: 156.3 loss_ce: 2.534 loss_giou: 2.752 loss_bbox: 17.19 loss_feat: 4.065 loss_dice: 0.7359 loss_rec: 36.19 loss_ce_0: 2.153 loss_giou_0: 2.023 loss_bbox_0: 5.972 loss_feat_0: 3.998 loss_dice_0: 0.7221 loss_ce_1: 2.277 loss_giou_1: 2.117 loss_bbox_1: 6.461 loss_feat_1: 4.216 loss_dice_1: 0.7619 loss_ce_2: 1.696 loss_giou_2: 2.121 loss_bbox_2: 6.934 loss_feat_2: 3.656 loss_dice_2: 0.6347 loss_ce_3: 3.07 loss_giou_3: 2.463 loss_bbox_3: 12.59 loss_feat_3: 3.797 loss_dice_3: 0.6062 loss_ce_4: 2.609 loss_giou_4: 2.416 loss_bbox_4: 10.98 loss_feat_4: 3.776 loss_dice_4: 0.6268 time: 0.5216 data_time: 0.0134 lr: 7.2025e-07 max_mem: 9163M [08/23 11:41:27 d2.utils.events]: eta: 0:08:12 iter: 39 total_loss: 106.5 loss_ce: 2.532 loss_giou: 2.424 loss_bbox: 3.239 loss_feat: 4.096 loss_dice: 0.686 loss_rec: 35.2 loss_ce_0: 1.99 loss_giou_0: 1.799 loss_bbox_0: 3.055 loss_feat_0: 3.838 loss_dice_0: 0.6499 loss_ce_1: 2.237 loss_giou_1: 1.889 loss_bbox_1: 2.41 loss_feat_1: 3.825 loss_dice_1: 0.6002 loss_ce_2: 1.447 loss_giou_2: 1.955 loss_bbox_2: 2.633 loss_feat_2: 3.902 loss_dice_2: 0.5499 loss_ce_3: 2.261 loss_giou_3: 2.086 loss_bbox_3: 3.301 loss_feat_3: 4.232 loss_dice_3: 0.6611 loss_ce_4: 2.009 loss_giou_4: 2.291 loss_bbox_4: 3.046 loss_feat_4: 3.825 loss_dice_4: 0.771 time: 0.5240 data_time: 0.0022 lr: 1.2153e-06 max_mem: 14227M [08/23 11:41:38 d2.utils.events]: eta: 0:08:10 iter: 59 total_loss: 97.33 loss_ce: 2.307 loss_giou: 2.039 loss_bbox: 1.711 loss_feat: 4.264 loss_dice: 0.8659 loss_rec: 33.7 loss_ce_0: 1.849 loss_giou_0: 1.767 loss_bbox_0: 1.893 loss_feat_0: 4.013 loss_dice_0: 0.7856 loss_ce_1: 1.411 loss_giou_1: 1.731 loss_bbox_1: 1.971 loss_feat_1: 3.995 loss_dice_1: 0.8162 loss_ce_2: 1.539 loss_giou_2: 1.843 loss_bbox_2: 1.74 loss_feat_2: 3.915 loss_dice_2: 0.7662 loss_ce_3: 1.618 loss_giou_3: 1.97 loss_bbox_3: 1.869 loss_feat_3: 3.972 loss_dice_3: 0.8788 loss_ce_4: 1.645 loss_giou_4: 2.075 loss_bbox_4: 1.908 loss_feat_4: 4.006 loss_dice_4: 0.9231 time: 0.5303 data_time: 0.0023 lr: 1.7103e-06 max_mem: 14227M [08/23 11:41:48 d2.utils.events]: eta: 0:07:56 iter: 79 total_loss: 87.78 loss_ce: 2.142 loss_giou: 1.999 loss_bbox: 1.855 loss_feat: 4.243 loss_dice: 0.6953 loss_rec: 31.52 loss_ce_0: 1.448 loss_giou_0: 1.517 loss_bbox_0: 1.569 loss_feat_0: 3.583 loss_dice_0: 0.6532 loss_ce_1: 1.18 loss_giou_1: 1.659 loss_bbox_1: 1.515 loss_feat_1: 3.696 loss_dice_1: 0.7618 loss_ce_2: 1.229 loss_giou_2: 1.736 loss_bbox_2: 1.483 loss_feat_2: 3.334 loss_dice_2: 0.6578 loss_ce_3: 1.309 loss_giou_3: 1.716 loss_bbox_3: 1.63 loss_feat_3: 3.545 loss_dice_3: 0.6394 loss_ce_4: 1.265 loss_giou_4: 1.917 loss_bbox_4: 1.684 loss_feat_4: 3.583 loss_dice_4: 0.7183 time: 0.5237 data_time: 0.0024 lr: 2.2053e-06 max_mem: 14227M [08/23 11:41:58 d2.utils.events]: eta: 0:07:38 iter: 99 total_loss: 81.28 loss_ce: 1.397 loss_giou: 2.129 loss_bbox: 1.673 loss_feat: 3.539 loss_dice: 0.6929 loss_rec: 28.82 loss_ce_0: 1.244 loss_giou_0: 1.749 loss_bbox_0: 1.271 loss_feat_0: 3.361 loss_dice_0: 0.6499 loss_ce_1: 1.279 loss_giou_1: 1.741 loss_bbox_1: 1.166 loss_feat_1: 3.502 loss_dice_1: 0.7502 loss_ce_2: 1.104 loss_giou_2: 1.635 loss_bbox_2: 1.005 loss_feat_2: 3.102 loss_dice_2: 0.711 loss_ce_3: 1.091 loss_giou_3: 1.871 loss_bbox_3: 1.615 loss_feat_3: 3.269 loss_dice_3: 0.5624 loss_ce_4: 1.086 loss_giou_4: 2.107 loss_bbox_4: 1.615 loss_feat_4: 3.709 loss_dice_4: 0.7628 time: 0.5163 data_time: 0.0023 lr: 2.7003e-06 max_mem: 14227M [08/23 11:42:08 d2.utils.events]: eta: 0:07:30 iter: 119 total_loss: 76.37 loss_ce: 1.029 loss_giou: 2.089 loss_bbox: 1.85 loss_feat: 3.039 loss_dice: 0.5508 loss_rec: 23.56 loss_ce_0: 1.45 loss_giou_0: 1.914 loss_bbox_0: 2.715 loss_feat_0: 2.946 loss_dice_0: 0.5873 loss_ce_1: 1.07 loss_giou_1: 1.691 loss_bbox_1: 2.166 loss_feat_1: 2.965 loss_dice_1: 0.5909 loss_ce_2: 1.213 loss_giou_2: 1.763 loss_bbox_2: 1.795 loss_feat_2: 2.872 loss_dice_2: 0.4652 loss_ce_3: 1.05 loss_giou_3: 1.938 loss_bbox_3: 2.043 loss_feat_3: 2.995 loss_dice_3: 0.5298 loss_ce_4: 1.05 loss_giou_4: 1.968 loss_bbox_4: 1.94 loss_feat_4: 3.097 loss_dice_4: 0.5557 time: 0.5158 data_time: 0.0025 lr: 3.1953e-06 max_mem: 14227M [08/23 11:42:19 d2.utils.events]: eta: 0:07:21 iter: 139 total_loss: 80.82 loss_ce: 1.24 loss_giou: 2.557 loss_bbox: 2.63 loss_feat: 3.233 loss_dice: 0.5434 loss_rec: 19.24 loss_ce_0: 1.17 loss_giou_0: 2.342 loss_bbox_0: 3.711 loss_feat_0: 2.799 loss_dice_0: 0.5648 loss_ce_1: 1.123 loss_giou_1: 2.318 loss_bbox_1: 3.263 loss_feat_1: 3.134 loss_dice_1: 0.667 loss_ce_2: 0.9466 loss_giou_2: 2.612 loss_bbox_2: 2.88 loss_feat_2: 2.914 loss_dice_2: 0.5317 loss_ce_3: 0.9809 loss_giou_3: 2.654 loss_bbox_3: 2.62 loss_feat_3: 3.041 loss_dice_3: 0.5635 loss_ce_4: 0.8791 loss_giou_4: 2.688 loss_bbox_4: 2.963 loss_feat_4: 3.052 loss_dice_4: 0.599 time: 0.5162 data_time: 0.0022 lr: 3.6903e-06 max_mem: 14227M [08/23 11:42:29 d2.utils.events]: eta: 0:07:08 iter: 159 total_loss: 76.54 loss_ce: 1.113 loss_giou: 2.13 loss_bbox: 1.466 loss_feat: 3.284 loss_dice: 0.7072 loss_rec: 17.14 loss_ce_0: 1.162 loss_giou_0: 2.605 loss_bbox_0: 4.531 loss_feat_0: 2.699 loss_dice_0: 0.6288 loss_ce_1: 1.283 loss_giou_1: 2.133 loss_bbox_1: 3.455 loss_feat_1: 3.234 loss_dice_1: 0.7173 loss_ce_2: 1.108 loss_giou_2: 2.366 loss_bbox_2: 2.372 loss_feat_2: 2.884 loss_dice_2: 0.5892 loss_ce_3: 1.088 loss_giou_3: 2.214 loss_bbox_3: 1.869 loss_feat_3: 2.946 loss_dice_3: 0.6167 loss_ce_4: 1.163 loss_giou_4: 2.285 loss_bbox_4: 1.883 loss_feat_4: 2.9 loss_dice_4: 0.5991 time: 0.5143 data_time: 0.0043 lr: 4.1853e-06 max_mem: 14227M [08/23 11:42:40 d2.utils.events]: eta: 0:06:59 iter: 179 total_loss: 104.2 loss_ce: 1.093 loss_giou: 2.904 loss_bbox: 3.882 loss_feat: 2.979 loss_dice: 0.5703 loss_rec: 14.11 loss_ce_0: 1.291 loss_giou_0: 3.215 loss_bbox_0: 10.61 loss_feat_0: 2.563 loss_dice_0: 0.4504 loss_ce_1: 1.182 loss_giou_1: 3.449 loss_bbox_1: 10.19 loss_feat_1: 3.103 loss_dice_1: 0.5367 loss_ce_2: 1.253 loss_giou_2: 3.293 loss_bbox_2: 8.522 loss_feat_2: 2.739 loss_dice_2: 0.5277 loss_ce_3: 1.655 loss_giou_3: 3.091 loss_bbox_3: 6.424 loss_feat_3: 2.94 loss_dice_3: 0.5103 loss_ce_4: 1.088 loss_giou_4: 2.963 loss_bbox_4: 5.075 loss_feat_4: 2.868 loss_dice_4: 0.5019 time: 0.5189 data_time: 0.0142 lr: 4.6802e-06 max_mem: 14227M [08/23 11:42:51 d2.utils.events]: eta: 0:06:50 iter: 199 total_loss: 73.38 loss_ce: 1.023 loss_giou: 2.118 loss_bbox: 1.68 loss_feat: 2.968 loss_dice: 0.5719 loss_rec: 13.52 loss_ce_0: 1.223 loss_giou_0: 2.717 loss_bbox_0: 6.557 loss_feat_0: 2.546 loss_dice_0: 0.6552 loss_ce_1: 1.19 loss_giou_1: 2.08 loss_bbox_1: 3.334 loss_feat_1: 3.23 loss_dice_1: 0.6504 loss_ce_2: 1.207 loss_giou_2: 1.877 loss_bbox_2: 1.798 loss_feat_2: 2.7 loss_dice_2: 0.6189 loss_ce_3: 1.141 loss_giou_3: 2.099 loss_bbox_3: 1.47 loss_feat_3: 3.042 loss_dice_3: 0.6933 loss_ce_4: 1.15 loss_giou_4: 2.022 loss_bbox_4: 1.587 loss_feat_4: 3.003 loss_dice_4: 0.6218 time: 0.5223 data_time: 0.0186 lr: 5.1753e-06 max_mem: 14227M [08/23 11:43:03 d2.utils.events]: eta: 0:06:45 iter: 219 total_loss: 67.84 loss_ce: 0.7925 loss_giou: 2.616 loss_bbox: 3.069 loss_feat: 2.719 loss_dice: 0.5113 loss_rec: 10.53 loss_ce_0: 1.047 loss_giou_0: 2.042 loss_bbox_0: 2.912 loss_feat_0: 2.266 loss_dice_0: 0.5216 loss_ce_1: 1.045 loss_giou_1: 2.183 loss_bbox_1: 2.922 loss_feat_1: 2.646 loss_dice_1: 0.6563 loss_ce_2: 1.053 loss_giou_2: 2.31 loss_bbox_2: 3.019 loss_feat_2: 2.584 loss_dice_2: 0.5102 loss_ce_3: 0.9405 loss_giou_3: 2.49 loss_bbox_3: 2.835 loss_feat_3: 2.625 loss_dice_3: 0.5153 loss_ce_4: 1.122 loss_giou_4: 2.503 loss_bbox_4: 2.651 loss_feat_4: 2.618 loss_dice_4: 0.5413 time: 0.5278 data_time: 0.0023 lr: 5.6703e-06 max_mem: 14227M [08/23 11:43:13 d2.utils.events]: eta: 0:06:34 iter: 239 total_loss: 71.62 loss_ce: 1.347 loss_giou: 2.101 loss_bbox: 2.872 loss_feat: 2.793 loss_dice: 0.5863 loss_rec: 8.665 loss_ce_0: 1.066 loss_giou_0: 2.691 loss_bbox_0: 5.064 loss_feat_0: 2.374 loss_dice_0: 0.5549 loss_ce_1: 1.077 loss_giou_1: 2.299 loss_bbox_1: 4.634 loss_feat_1: 2.623 loss_dice_1: 0.6471 loss_ce_2: 1.057 loss_giou_2: 1.855 loss_bbox_2: 2.942 loss_feat_2: 2.594 loss_dice_2: 0.6508 loss_ce_3: 1.036 loss_giou_3: 2.032 loss_bbox_3: 3.045 loss_feat_3: 2.785 loss_dice_3: 0.565 loss_ce_4: 1.427 loss_giou_4: 2.286 loss_bbox_4: 3.092 loss_feat_4: 2.936 loss_dice_4: 0.6067 time: 0.5276 data_time: 0.0029 lr: 6.1652e-06 max_mem: 14227M [08/23 11:43:23 d2.utils.events]: eta: 0:06:23 iter: 259 total_loss: 60.39 loss_ce: 1.039 loss_giou: 2.217 loss_bbox: 2.463 loss_feat: 2.434 loss_dice: 0.5098 loss_rec: 7.374 loss_ce_0: 2.444 loss_giou_0: 2.069 loss_bbox_0: 4.134 loss_feat_0: 2.249 loss_dice_0: 0.4854 loss_ce_1: 1.404 loss_giou_1: 1.95 loss_bbox_1: 2.074 loss_feat_1: 2.555 loss_dice_1: 0.5805 loss_ce_2: 1 loss_giou_2: 2.014 loss_bbox_2: 2.017 loss_feat_2: 2.463 loss_dice_2: 0.4657 loss_ce_3: 0.9797 loss_giou_3: 2.374 loss_bbox_3: 2.188 loss_feat_3: 2.415 loss_dice_3: 0.5382 loss_ce_4: 1.101 loss_giou_4: 2.73 loss_bbox_4: 1.912 loss_feat_4: 2.283 loss_dice_4: 0.4874 time: 0.5262 data_time: 0.0023 lr: 6.6603e-06 max_mem: 14227M [08/23 11:43:34 d2.utils.events]: eta: 0:06:12 iter: 279 total_loss: 64.04 loss_ce: 1.01 loss_giou: 2.276 loss_bbox: 2.642 loss_feat: 2.255 loss_dice: 0.4168 loss_rec: 7.262 loss_ce_0: 1.456 loss_giou_0: 2.24 loss_bbox_0: 2.393 loss_feat_0: 2.075 loss_dice_0: 0.4259 loss_ce_1: 1.136 loss_giou_1: 2.532 loss_bbox_1: 2.557 loss_feat_1: 2.259 loss_dice_1: 0.5146 loss_ce_2: 1.009 loss_giou_2: 2.45 loss_bbox_2: 3.449 loss_feat_2: 2.323 loss_dice_2: 0.4469 loss_ce_3: 1.051 loss_giou_3: 2.727 loss_bbox_3: 3.667 loss_feat_3: 2.222 loss_dice_3: 0.3848 loss_ce_4: 0.9264 loss_giou_4: 2.715 loss_bbox_4: 2.423 loss_feat_4: 2.456 loss_dice_4: 0.427 time: 0.5247 data_time: 0.0023 lr: 7.1553e-06 max_mem: 14227M [08/23 11:43:43 d2.utils.events]: eta: 0:05:59 iter: 299 total_loss: 69.41 loss_ce: 1.197 loss_giou: 2.196 loss_bbox: 1.933 loss_feat: 2.464 loss_dice: 0.6606 loss_rec: 6.806 loss_ce_0: 1.057 loss_giou_0: 2.41 loss_bbox_0: 4.458 loss_feat_0: 2.171 loss_dice_0: 0.5951 loss_ce_1: 1.144 loss_giou_1: 2.643 loss_bbox_1: 5.074 loss_feat_1: 2.826 loss_dice_1: 0.6552 loss_ce_2: 1.137 loss_giou_2: 2.716 loss_bbox_2: 2.962 loss_feat_2: 2.362 loss_dice_2: 0.5416 loss_ce_3: 0.9044 loss_giou_3: 2.752 loss_bbox_3: 3.33 loss_feat_3: 2.322 loss_dice_3: 0.6127 loss_ce_4: 1.253 loss_giou_4: 2.177 loss_bbox_4: 2.118 loss_feat_4: 2.413 loss_dice_4: 0.6622 time: 0.5223 data_time: 0.0022 lr: 7.6503e-06 max_mem: 14227M [08/23 11:43:53 d2.utils.events]: eta: 0:05:47 iter: 319 total_loss: 198.2 loss_ce: 2.517 loss_giou: 3.912 loss_bbox: 19.54 loss_feat: 2.619 loss_dice: 0.6295 loss_rec: 7.498 loss_ce_0: 1.543 loss_giou_0: 3.841 loss_bbox_0: 25.12 loss_feat_0: 2.19 loss_dice_0: 0.5113 loss_ce_1: 1.172 loss_giou_1: 3.885 loss_bbox_1: 21.6 loss_feat_1: 2.504 loss_dice_1: 0.5296 loss_ce_2: 1.1 loss_giou_2: 3.915 loss_bbox_2: 17.08 loss_feat_2: 2.375 loss_dice_2: 0.5556 loss_ce_3: 1.301 loss_giou_3: 3.898 loss_bbox_3: 18.16 loss_feat_3: 2.317 loss_dice_3: 0.5113 loss_ce_4: 1.095 loss_giou_4: 3.889 loss_bbox_4: 19.19 loss_feat_4: 2.62 loss_dice_4: 0.5921 time: 0.5204 data_time: 0.0022 lr: 8.1452e-06 max_mem: 14227M [08/23 11:44:03 d2.utils.events]: eta: 0:05:36 iter: 339 total_loss: 153.4 loss_ce: 1.18 loss_giou: 3.787 loss_bbox: 14.62 loss_feat: 2.174 loss_dice: 0.4536 loss_rec: 6.117 loss_ce_0: 1.497 loss_giou_0: 3.946 loss_bbox_0: 26.5 loss_feat_0: 2.262 loss_dice_0: 0.5027 loss_ce_1: 1.039 loss_giou_1: 3.782 loss_bbox_1: 14.23 loss_feat_1: 2.592 loss_dice_1: 0.54 loss_ce_2: 1.076 loss_giou_2: 3.785 loss_bbox_2: 14.52 loss_feat_2: 2.245 loss_dice_2: 0.5031 loss_ce_3: 1.007 loss_giou_3: 3.786 loss_bbox_3: 14.65 loss_feat_3: 2.357 loss_dice_3: 0.4409 loss_ce_4: 0.9694 loss_giou_4: 3.787 loss_bbox_4: 14.82 loss_feat_4: 2.333 loss_dice_4: 0.5093 time: 0.5185 data_time: 0.0022 lr: 8.6403e-06 max_mem: 14227M [08/23 11:44:15 d2.utils.events]: eta: 0:05:28 iter: 359 total_loss: 76.52 loss_ce: 1.054 loss_giou: 3.003 loss_bbox: 4.644 loss_feat: 2.053 loss_dice: 0.5033 loss_rec: 6.769 loss_ce_0: 1.09 loss_giou_0: 2.685 loss_bbox_0: 3.354 loss_feat_0: 2.091 loss_dice_0: 0.4582 loss_ce_1: 1.036 loss_giou_1: 2.788 loss_bbox_1: 4.534 loss_feat_1: 2.272 loss_dice_1: 0.5629 loss_ce_2: 1.152 loss_giou_2: 3.065 loss_bbox_2: 5.047 loss_feat_2: 2.082 loss_dice_2: 0.5167 loss_ce_3: 1.047 loss_giou_3: 3.288 loss_bbox_3: 5.003 loss_feat_3: 2.155 loss_dice_3: 0.4854 loss_ce_4: 0.9994 loss_giou_4: 3.495 loss_bbox_4: 6.143 loss_feat_4: 2.206 loss_dice_4: 0.4923 time: 0.5214 data_time: 0.0267 lr: 9.1353e-06 max_mem: 14227M [08/23 11:44:25 d2.utils.events]: eta: 0:05:17 iter: 379 total_loss: 83.61 loss_ce: 1.163 loss_giou: 2.897 loss_bbox: 5.277 loss_feat: 2.139 loss_dice: 0.4408 loss_rec: 6.365 loss_ce_0: 1.111 loss_giou_0: 3.71 loss_bbox_0: 10.77 loss_feat_0: 1.963 loss_dice_0: 0.5216 loss_ce_1: 1.058 loss_giou_1: 3.673 loss_bbox_1: 9.577 loss_feat_1: 2.194 loss_dice_1: 0.4702 loss_ce_2: 1.09 loss_giou_2: 3.418 loss_bbox_2: 5.111 loss_feat_2: 2.153 loss_dice_2: 0.4884 loss_ce_3: 0.9303 loss_giou_3: 2.859 loss_bbox_3: 4.919 loss_feat_3: 2.199 loss_dice_3: 0.4309 loss_ce_4: 1.039 loss_giou_4: 2.868 loss_bbox_4: 5.215 loss_feat_4: 2.195 loss_dice_4: 0.4768 time: 0.5205 data_time: 0.0025 lr: 9.6302e-06 max_mem: 14227M [08/23 11:44:34 d2.utils.events]: eta: 0:05:05 iter: 399 total_loss: 73.44 loss_ce: 1.275 loss_giou: 2.651 loss_bbox: 3.799 loss_feat: 1.971 loss_dice: 0.5119 loss_rec: 5.403 loss_ce_0: 1.083 loss_giou_0: 2.534 loss_bbox_0: 4.132 loss_feat_0: 2.023 loss_dice_0: 0.4785 loss_ce_1: 1.047 loss_giou_1: 2.968 loss_bbox_1: 4.696 loss_feat_1: 2.194 loss_dice_1: 0.544 loss_ce_2: 1.052 loss_giou_2: 3.018 loss_bbox_2: 4.164 loss_feat_2: 2.216 loss_dice_2: 0.5035 loss_ce_3: 1.641 loss_giou_3: 3.069 loss_bbox_3: 4.359 loss_feat_3: 2.035 loss_dice_3: 0.4639 loss_ce_4: 1.597 loss_giou_4: 3.014 loss_bbox_4: 4.134 loss_feat_4: 2.034 loss_dice_4: 0.5006 time: 0.5185 data_time: 0.0023 lr: 1.0125e-05 max_mem: 14227M [08/23 11:44:45 d2.utils.events]: eta: 0:04:55 iter: 419 total_loss: 59.57 loss_ce: 0.9784 loss_giou: 2.729 loss_bbox: 3.124 loss_feat: 1.903 loss_dice: 0.5214 loss_rec: 5.231 loss_ce_0: 1.046 loss_giou_0: 1.864 loss_bbox_0: 2.074 loss_feat_0: 2.01 loss_dice_0: 0.4988 loss_ce_1: 0.9948 loss_giou_1: 2.218 loss_bbox_1: 3.033 loss_feat_1: 2.049 loss_dice_1: 0.5673 loss_ce_2: 1.192 loss_giou_2: 2.26 loss_bbox_2: 2.938 loss_feat_2: 1.993 loss_dice_2: 0.4993 loss_ce_3: 1.128 loss_giou_3: 2.677 loss_bbox_3: 3.033 loss_feat_3: 1.965 loss_dice_3: 0.524 loss_ce_4: 1.119 loss_giou_4: 2.73 loss_bbox_4: 3.036 loss_feat_4: 1.856 loss_dice_4: 0.5318 time: 0.5186 data_time: 0.0024 lr: 1.062e-05 max_mem: 14227M [08/23 11:44:55 d2.utils.events]: eta: 0:04:45 iter: 439 total_loss: 60.74 loss_ce: 0.9373 loss_giou: 2.597 loss_bbox: 3.071 loss_feat: 1.909 loss_dice: 0.5605 loss_rec: 4.985 loss_ce_0: 0.991 loss_giou_0: 2.802 loss_bbox_0: 4.416 loss_feat_0: 2.009 loss_dice_0: 0.5286 loss_ce_1: 1.06 loss_giou_1: 2.392 loss_bbox_1: 2.99 loss_feat_1: 2.117 loss_dice_1: 0.591 loss_ce_2: 1.21 loss_giou_2: 2.69 loss_bbox_2: 3.225 loss_feat_2: 2.007 loss_dice_2: 0.5651 loss_ce_3: 0.973 loss_giou_3: 2.656 loss_bbox_3: 3.985 loss_feat_3: 1.927 loss_dice_3: 0.5281 loss_ce_4: 1.019 loss_giou_4: 2.574 loss_bbox_4: 3.501 loss_feat_4: 1.918 loss_dice_4: 0.5419 time: 0.5180 data_time: 0.0024 lr: 1.1115e-05 max_mem: 14227M [08/23 11:45:04 d2.utils.events]: eta: 0:04:33 iter: 459 total_loss: 108.6 loss_ce: 1.081 loss_giou: 3.767 loss_bbox: 10.59 loss_feat: 2.044 loss_dice: 0.5425 loss_rec: 4.824 loss_ce_0: 0.9782 loss_giou_0: 3.354 loss_bbox_0: 7.549 loss_feat_0: 2.028 loss_dice_0: 0.4745 loss_ce_1: 1.037 loss_giou_1: 3.661 loss_bbox_1: 9.298 loss_feat_1: 2.199 loss_dice_1: 0.5402 loss_ce_2: 1.25 loss_giou_2: 3.683 loss_bbox_2: 10.21 loss_feat_2: 2.077 loss_dice_2: 0.5451 loss_ce_3: 1.204 loss_giou_3: 3.753 loss_bbox_3: 9.802 loss_feat_3: 2.045 loss_dice_3: 0.5314 loss_ce_4: 1.108 loss_giou_4: 3.792 loss_bbox_4: 10.39 loss_feat_4: 1.968 loss_dice_4: 0.4995 time: 0.5158 data_time: 0.0022 lr: 1.161e-05 max_mem: 14227M [08/23 11:45:15 d2.utils.events]: eta: 0:04:23 iter: 479 total_loss: 62.96 loss_ce: 0.9463 loss_giou: 2.657 loss_bbox: 3.848 loss_feat: 2.244 loss_dice: 0.5498 loss_rec: 5.251 loss_ce_0: 1.043 loss_giou_0: 2.364 loss_bbox_0: 3.548 loss_feat_0: 1.983 loss_dice_0: 0.4368 loss_ce_1: 0.9378 loss_giou_1: 2.137 loss_bbox_1: 3.009 loss_feat_1: 2.089 loss_dice_1: 0.5575 loss_ce_2: 0.9831 loss_giou_2: 2.848 loss_bbox_2: 3.922 loss_feat_2: 2.007 loss_dice_2: 0.4493 loss_ce_3: 0.9369 loss_giou_3: 2.667 loss_bbox_3: 2.834 loss_feat_3: 1.978 loss_dice_3: 0.5477 loss_ce_4: 0.8923 loss_giou_4: 2.939 loss_bbox_4: 5.538 loss_feat_4: 2.014 loss_dice_4: 0.5224 time: 0.5162 data_time: 0.0024 lr: 1.2105e-05 max_mem: 14227M [08/23 11:45:25 d2.utils.events]: eta: 0:04:13 iter: 499 total_loss: 56.16 loss_ce: 1.005 loss_giou: 2.834 loss_bbox: 3.792 loss_feat: 1.96 loss_dice: 0.452 loss_rec: 4.706 loss_ce_0: 1.11 loss_giou_0: 1.92 loss_bbox_0: 2.129 loss_feat_0: 1.879 loss_dice_0: 0.4258 loss_ce_1: 1.084 loss_giou_1: 1.842 loss_bbox_1: 2.107 loss_feat_1: 1.894 loss_dice_1: 0.4559 loss_ce_2: 1.323 loss_giou_2: 2.028 loss_bbox_2: 2.919 loss_feat_2: 1.879 loss_dice_2: 0.4232 loss_ce_3: 1.089 loss_giou_3: 2.174 loss_bbox_3: 2.791 loss_feat_3: 2.043 loss_dice_3: 0.4292 loss_ce_4: 1.059 loss_giou_4: 2.352 loss_bbox_4: 2.88 loss_feat_4: 2.136 loss_dice_4: 0.5225 time: 0.5154 data_time: 0.0024 lr: 1.26e-05 max_mem: 14227M [08/23 11:45:35 d2.utils.events]: eta: 0:04:01 iter: 519 total_loss: 91.45 loss_ce: 1.416 loss_giou: 2.79 loss_bbox: 4.732 loss_feat: 2.204 loss_dice: 0.5733 loss_rec: 4.868 loss_ce_0: 1.102 loss_giou_0: 3.042 loss_bbox_0: 6.121 loss_feat_0: 1.982 loss_dice_0: 0.5326 loss_ce_1: 1.339 loss_giou_1: 2.922 loss_bbox_1: 6.383 loss_feat_1: 2.334 loss_dice_1: 0.6091 loss_ce_2: 2.074 loss_giou_2: 2.128 loss_bbox_2: 3.305 loss_feat_2: 2.978 loss_dice_2: 0.6573 loss_ce_3: 0.9298 loss_giou_3: 2.258 loss_bbox_3: 5.988 loss_feat_3: 2.524 loss_dice_3: 0.6021 loss_ce_4: 1.353 loss_giou_4: 3.429 loss_bbox_4: 11.32 loss_feat_4: 2.477 loss_dice_4: 0.6743 time: 0.5145 data_time: 0.0023 lr: 1.3095e-05 max_mem: 14227M [08/23 11:45:45 d2.utils.events]: eta: 0:03:52 iter: 539 total_loss: 64.66 loss_ce: 1.047 loss_giou: 2.95 loss_bbox: 3.51 loss_feat: 2.12 loss_dice: 0.58 loss_rec: 5.259 loss_ce_0: 1.023 loss_giou_0: 2.766 loss_bbox_0: 6.499 loss_feat_0: 1.888 loss_dice_0: 0.5637 loss_ce_1: 1.564 loss_giou_1: 1.803 loss_bbox_1: 1.737 loss_feat_1: 2.328 loss_dice_1: 0.5707 loss_ce_2: 1.31 loss_giou_2: 2.063 loss_bbox_2: 1.936 loss_feat_2: 2.253 loss_dice_2: 0.5629 loss_ce_3: 0.7893 loss_giou_3: 2.318 loss_bbox_3: 2.62 loss_feat_3: 2.013 loss_dice_3: 0.5631 loss_ce_4: 0.9379 loss_giou_4: 2.909 loss_bbox_4: 5.444 loss_feat_4: 2.135 loss_dice_4: 0.5592 time: 0.5147 data_time: 0.0064 lr: 1.359e-05 max_mem: 14227M [08/23 11:45:57 d2.utils.events]: eta: 0:03:43 iter: 559 total_loss: 58.25 loss_ce: 1.001 loss_giou: 2.906 loss_bbox: 2.602 loss_feat: 2.008 loss_dice: 0.4961 loss_rec: 5.3 loss_ce_0: 0.9743 loss_giou_0: 1.773 loss_bbox_0: 1.775 loss_feat_0: 1.858 loss_dice_0: 0.4611 loss_ce_1: 1.051 loss_giou_1: 1.924 loss_bbox_1: 1.808 loss_feat_1: 1.92 loss_dice_1: 0.4668 loss_ce_2: 1.181 loss_giou_2: 2.42 loss_bbox_2: 2.717 loss_feat_2: 2.001 loss_dice_2: 0.4607 loss_ce_3: 0.8418 loss_giou_3: 2.268 loss_bbox_3: 2.933 loss_feat_3: 1.894 loss_dice_3: 0.5083 loss_ce_4: 0.8543 loss_giou_4: 2.75 loss_bbox_4: 5.999 loss_feat_4: 1.963 loss_dice_4: 0.4846 time: 0.5174 data_time: 0.0023 lr: 1.4085e-05 max_mem: 14227M [08/23 11:46:07 d2.utils.events]: eta: 0:03:33 iter: 579 total_loss: 63.69 loss_ce: 1.053 loss_giou: 2.579 loss_bbox: 4.553 loss_feat: 2.119 loss_dice: 0.6019 loss_rec: 5.444 loss_ce_0: 1.11 loss_giou_0: 1.389 loss_bbox_0: 1.582 loss_feat_0: 2.144 loss_dice_0: 0.5904 loss_ce_1: 1.096 loss_giou_1: 1.535 loss_bbox_1: 2.254 loss_feat_1: 2.181 loss_dice_1: 0.6212 loss_ce_2: 1.144 loss_giou_2: 1.775 loss_bbox_2: 2.392 loss_feat_2: 2.102 loss_dice_2: 0.6043 loss_ce_3: 1.31 loss_giou_3: 2.296 loss_bbox_3: 3.727 loss_feat_3: 2.168 loss_dice_3: 0.6228 loss_ce_4: 1.342 loss_giou_4: 2.357 loss_bbox_4: 4.392 loss_feat_4: 2.153 loss_dice_4: 0.6067 time: 0.5164 data_time: 0.0023 lr: 1.458e-05 max_mem: 14227M [08/23 11:46:17 d2.utils.events]: eta: 0:03:23 iter: 599 total_loss: 53.64 loss_ce: 0.9775 loss_giou: 2.34 loss_bbox: 2.363 loss_feat: 1.844 loss_dice: 0.5029 loss_rec: 4.306 loss_ce_0: 1.089 loss_giou_0: 1.963 loss_bbox_0: 1.856 loss_feat_0: 1.921 loss_dice_0: 0.4897 loss_ce_1: 1.076 loss_giou_1: 2.15 loss_bbox_1: 2.273 loss_feat_1: 1.919 loss_dice_1: 0.5088 loss_ce_2: 0.971 loss_giou_2: 2.205 loss_bbox_2: 2.306 loss_feat_2: 1.908 loss_dice_2: 0.5314 loss_ce_3: 0.9614 loss_giou_3: 2.168 loss_bbox_3: 2.556 loss_feat_3: 1.909 loss_dice_3: 0.5187 loss_ce_4: 0.9768 loss_giou_4: 2.344 loss_bbox_4: 3.022 loss_feat_4: 1.892 loss_dice_4: 0.5066 time: 0.5159 data_time: 0.0022 lr: 1.5075e-05 max_mem: 14227M [08/23 11:46:27 d2.utils.events]: eta: 0:03:12 iter: 619 total_loss: 57.99 loss_ce: 0.8259 loss_giou: 2.905 loss_bbox: 3.446 loss_feat: 2.026 loss_dice: 0.6076 loss_rec: 5.115 loss_ce_0: 1.027 loss_giou_0: 1.927 loss_bbox_0: 1.314 loss_feat_0: 1.959 loss_dice_0: 0.5615 loss_ce_1: 1.095 loss_giou_1: 2.106 loss_bbox_1: 1.316 loss_feat_1: 1.971 loss_dice_1: 0.5789 loss_ce_2: 0.9852 loss_giou_2: 1.9 loss_bbox_2: 1.799 loss_feat_2: 2.021 loss_dice_2: 0.6376 loss_ce_3: 0.9696 loss_giou_3: 1.978 loss_bbox_3: 2.592 loss_feat_3: 1.948 loss_dice_3: 0.5976 loss_ce_4: 0.8398 loss_giou_4: 2.677 loss_bbox_4: 3.229 loss_feat_4: 1.991 loss_dice_4: 0.5934 time: 0.5156 data_time: 0.0023 lr: 1.557e-05 max_mem: 14227M [08/23 11:46:38 d2.utils.events]: eta: 0:03:02 iter: 639 total_loss: 57.14 loss_ce: 1.024 loss_giou: 2.912 loss_bbox: 3.251 loss_feat: 1.962 loss_dice: 0.5381 loss_rec: 4.61 loss_ce_0: 1.046 loss_giou_0: 1.673 loss_bbox_0: 2.087 loss_feat_0: 1.983 loss_dice_0: 0.5732 loss_ce_1: 1.104 loss_giou_1: 1.968 loss_bbox_1: 2.332 loss_feat_1: 1.95 loss_dice_1: 0.5765 loss_ce_2: 0.9936 loss_giou_2: 2.134 loss_bbox_2: 2.655 loss_feat_2: 1.94 loss_dice_2: 0.5647 loss_ce_3: 1.059 loss_giou_3: 2.064 loss_bbox_3: 2.282 loss_feat_3: 1.895 loss_dice_3: 0.5646 loss_ce_4: 1.056 loss_giou_4: 2.7 loss_bbox_4: 2.836 loss_feat_4: 1.987 loss_dice_4: 0.5369 time: 0.5162 data_time: 0.0272 lr: 1.6065e-05 max_mem: 14227M [08/23 11:46:48 d2.utils.events]: eta: 0:02:52 iter: 659 total_loss: 77.89 loss_ce: 1.281 loss_giou: 3.073 loss_bbox: 4.905 loss_feat: 1.981 loss_dice: 0.5196 loss_rec: 4.796 loss_ce_0: 1.122 loss_giou_0: 1.988 loss_bbox_0: 2.863 loss_feat_0: 1.983 loss_dice_0: 0.5145 loss_ce_1: 1.128 loss_giou_1: 2.231 loss_bbox_1: 4.387 loss_feat_1: 2.055 loss_dice_1: 0.5172 loss_ce_2: 1.183 loss_giou_2: 2.728 loss_bbox_2: 5.112 loss_feat_2: 1.981 loss_dice_2: 0.5328 loss_ce_3: 1.335 loss_giou_3: 3.105 loss_bbox_3: 7.269 loss_feat_3: 2.154 loss_dice_3: 0.5609 loss_ce_4: 1.247 loss_giou_4: 3.359 loss_bbox_4: 7.992 loss_feat_4: 1.975 loss_dice_4: 0.5009 time: 0.5166 data_time: 0.0042 lr: 1.656e-05 max_mem: 14227M [08/23 11:47:00 d2.utils.events]: eta: 0:02:43 iter: 679 total_loss: 55.49 loss_ce: 0.9883 loss_giou: 2.728 loss_bbox: 2.465 loss_feat: 1.921 loss_dice: 0.5063 loss_rec: 4.521 loss_ce_0: 1.06 loss_giou_0: 1.896 loss_bbox_0: 1.773 loss_feat_0: 1.891 loss_dice_0: 0.4641 loss_ce_1: 1.089 loss_giou_1: 1.977 loss_bbox_1: 2.094 loss_feat_1: 1.888 loss_dice_1: 0.4765 loss_ce_2: 1.068 loss_giou_2: 1.97 loss_bbox_2: 2.291 loss_feat_2: 1.874 loss_dice_2: 0.4692 loss_ce_3: 1.003 loss_giou_3: 2.287 loss_bbox_3: 3.679 loss_feat_3: 1.904 loss_dice_3: 0.4966 loss_ce_4: 1.012 loss_giou_4: 2.558 loss_bbox_4: 2.964 loss_feat_4: 1.963 loss_dice_4: 0.5 time: 0.5191 data_time: 0.0238 lr: 1.7055e-05 max_mem: 14227M [08/23 11:47:11 d2.utils.events]: eta: 0:02:33 iter: 699 total_loss: 58.55 loss_ce: 1.107 loss_giou: 2.477 loss_bbox: 2.593 loss_feat: 1.941 loss_dice: 0.5284 loss_rec: 4.572 loss_ce_0: 1.088 loss_giou_0: 1.791 loss_bbox_0: 2.329 loss_feat_0: 2.02 loss_dice_0: 0.5717 loss_ce_1: 1.135 loss_giou_1: 2.181 loss_bbox_1: 2.522 loss_feat_1: 2.006 loss_dice_1: 0.5317 loss_ce_2: 1.079 loss_giou_2: 2.44 loss_bbox_2: 2.93 loss_feat_2: 1.945 loss_dice_2: 0.498 loss_ce_3: 1.123 loss_giou_3: 2.685 loss_bbox_3: 2.692 loss_feat_3: 1.942 loss_dice_3: 0.5253 loss_ce_4: 1.115 loss_giou_4: 2.555 loss_bbox_4: 2.648 loss_feat_4: 1.932 loss_dice_4: 0.536 time: 0.5199 data_time: 0.0024 lr: 1.755e-05 max_mem: 14227M [08/23 11:47:21 d2.utils.events]: eta: 0:02:22 iter: 719 total_loss: 45.81 loss_ce: 0.9543 loss_giou: 2.159 loss_bbox: 1.653 loss_feat: 1.867 loss_dice: 0.4913 loss_rec: 4.448 loss_ce_0: 0.9851 loss_giou_0: 1.748 loss_bbox_0: 1.689 loss_feat_0: 1.863 loss_dice_0: 0.4042 loss_ce_1: 1.038 loss_giou_1: 1.754 loss_bbox_1: 1.386 loss_feat_1: 1.834 loss_dice_1: 0.4395 loss_ce_2: 1.004 loss_giou_2: 1.984 loss_bbox_2: 1.555 loss_feat_2: 1.851 loss_dice_2: 0.514 loss_ce_3: 0.9951 loss_giou_3: 2.127 loss_bbox_3: 1.66 loss_feat_3: 1.882 loss_dice_3: 0.4754 loss_ce_4: 0.997 loss_giou_4: 2.129 loss_bbox_4: 1.713 loss_feat_4: 1.785 loss_dice_4: 0.4549 time: 0.5192 data_time: 0.0023 lr: 1.8045e-05 max_mem: 14227M [08/23 11:47:32 d2.utils.events]: eta: 0:02:12 iter: 739 total_loss: 51.59 loss_ce: 0.9737 loss_giou: 2.829 loss_bbox: 1.919 loss_feat: 1.941 loss_dice: 0.4595 loss_rec: 4.418 loss_ce_0: 1.046 loss_giou_0: 1.78 loss_bbox_0: 1.364 loss_feat_0: 1.911 loss_dice_0: 0.4611 loss_ce_1: 1.041 loss_giou_1: 1.855 loss_bbox_1: 2.01 loss_feat_1: 1.925 loss_dice_1: 0.4675 loss_ce_2: 1.153 loss_giou_2: 1.987 loss_bbox_2: 1.854 loss_feat_2: 1.931 loss_dice_2: 0.466 loss_ce_3: 0.9925 loss_giou_3: 2.303 loss_bbox_3: 2.063 loss_feat_3: 1.917 loss_dice_3: 0.4657 loss_ce_4: 0.998 loss_giou_4: 2.626 loss_bbox_4: 1.897 loss_feat_4: 1.931 loss_dice_4: 0.4439 time: 0.5191 data_time: 0.0024 lr: 1.854e-05 max_mem: 14227M [08/23 11:47:43 d2.utils.events]: eta: 0:02:02 iter: 759 total_loss: 47.6 loss_ce: 0.955 loss_giou: 2.486 loss_bbox: 2.144 loss_feat: 1.824 loss_dice: 0.3722 loss_rec: 3.777 loss_ce_0: 1.093 loss_giou_0: 1.752 loss_bbox_0: 1.805 loss_feat_0: 1.875 loss_dice_0: 0.3792 loss_ce_1: 0.9499 loss_giou_1: 1.808 loss_bbox_1: 1.686 loss_feat_1: 1.842 loss_dice_1: 0.3803 loss_ce_2: 1.136 loss_giou_2: 1.713 loss_bbox_2: 1.493 loss_feat_2: 1.854 loss_dice_2: 0.3976 loss_ce_3: 0.9008 loss_giou_3: 2.095 loss_bbox_3: 1.915 loss_feat_3: 1.883 loss_dice_3: 0.3768 loss_ce_4: 0.9757 loss_giou_4: 2.276 loss_bbox_4: 1.79 loss_feat_4: 1.825 loss_dice_4: 0.3821 time: 0.5198 data_time: 0.0050 lr: 1.9035e-05 max_mem: 14227M [08/23 11:47:54 d2.utils.events]: eta: 0:01:53 iter: 779 total_loss: 45.42 loss_ce: 0.9424 loss_giou: 2.084 loss_bbox: 1.809 loss_feat: 1.822 loss_dice: 0.4215 loss_rec: 4.492 loss_ce_0: 1.066 loss_giou_0: 1.513 loss_bbox_0: 1.471 loss_feat_0: 1.872 loss_dice_0: 0.4183 loss_ce_1: 1.013 loss_giou_1: 1.479 loss_bbox_1: 1.232 loss_feat_1: 1.837 loss_dice_1: 0.4034 loss_ce_2: 1.206 loss_giou_2: 1.707 loss_bbox_2: 1.539 loss_feat_2: 1.845 loss_dice_2: 0.4334 loss_ce_3: 0.9359 loss_giou_3: 2.019 loss_bbox_3: 2.075 loss_feat_3: 1.871 loss_dice_3: 0.4323 loss_ce_4: 0.9728 loss_giou_4: 1.981 loss_bbox_4: 1.825 loss_feat_4: 1.825 loss_dice_4: 0.4153 time: 0.5214 data_time: 0.0024 lr: 1.953e-05 max_mem: 14227M [08/23 11:48:05 d2.utils.events]: eta: 0:01:42 iter: 799 total_loss: 50.67 loss_ce: 1.009 loss_giou: 2.438 loss_bbox: 2.146 loss_feat: 1.916 loss_dice: 0.4311 loss_rec: 4.517 loss_ce_0: 1.046 loss_giou_0: 1.887 loss_bbox_0: 1.321 loss_feat_0: 1.945 loss_dice_0: 0.4248 loss_ce_1: 1.022 loss_giou_1: 1.839 loss_bbox_1: 1.675 loss_feat_1: 1.955 loss_dice_1: 0.4351 loss_ce_2: 1.091 loss_giou_2: 2.274 loss_bbox_2: 1.645 loss_feat_2: 1.959 loss_dice_2: 0.4251 loss_ce_3: 1.072 loss_giou_3: 2.507 loss_bbox_3: 2.081 loss_feat_3: 1.95 loss_dice_3: 0.4543 loss_ce_4: 1.057 loss_giou_4: 2.471 loss_bbox_4: 2.21 loss_feat_4: 1.962 loss_dice_4: 0.4912 time: 0.5218 data_time: 0.0023 lr: 2.0025e-05 max_mem: 14227M [08/23 11:48:17 d2.utils.events]: eta: 0:01:32 iter: 819 total_loss: 44.97 loss_ce: 0.8956 loss_giou: 2.115 loss_bbox: 1.454 loss_feat: 1.838 loss_dice: 0.4864 loss_rec: 4.243 loss_ce_0: 0.9723 loss_giou_0: 1.821 loss_bbox_0: 1.42 loss_feat_0: 1.831 loss_dice_0: 0.4908 loss_ce_1: 0.9698 loss_giou_1: 1.736 loss_bbox_1: 1.326 loss_feat_1: 1.812 loss_dice_1: 0.4722 loss_ce_2: 1.041 loss_giou_2: 1.888 loss_bbox_2: 1.302 loss_feat_2: 1.832 loss_dice_2: 0.4788 loss_ce_3: 0.9596 loss_giou_3: 1.879 loss_bbox_3: 1.293 loss_feat_3: 1.817 loss_dice_3: 0.493 loss_ce_4: 0.8982 loss_giou_4: 1.879 loss_bbox_4: 1.405 loss_feat_4: 1.865 loss_dice_4: 0.4981 time: 0.5230 data_time: 0.0179 lr: 2.052e-05 max_mem: 14227M [08/23 11:48:27 d2.utils.events]: eta: 0:01:22 iter: 839 total_loss: 48.88 loss_ce: 0.9432 loss_giou: 2.283 loss_bbox: 2.077 loss_feat: 1.897 loss_dice: 0.5156 loss_rec: 3.889 loss_ce_0: 1.001 loss_giou_0: 1.942 loss_bbox_0: 1.588 loss_feat_0: 1.905 loss_dice_0: 0.5007 loss_ce_1: 1.04 loss_giou_1: 2.029 loss_bbox_1: 1.755 loss_feat_1: 1.952 loss_dice_1: 0.5069 loss_ce_2: 1.02 loss_giou_2: 1.922 loss_bbox_2: 1.928 loss_feat_2: 1.845 loss_dice_2: 0.4907 loss_ce_3: 0.9729 loss_giou_3: 2.131 loss_bbox_3: 1.95 loss_feat_3: 1.848 loss_dice_3: 0.4915 loss_ce_4: 0.953 loss_giou_4: 1.939 loss_bbox_4: 1.778 loss_feat_4: 1.885 loss_dice_4: 0.5087 time: 0.5231 data_time: 0.0023 lr: 2.1015e-05 max_mem: 14227M [08/23 11:48:37 d2.utils.events]: eta: 0:01:12 iter: 859 total_loss: 48.5 loss_ce: 0.948 loss_giou: 2.026 loss_bbox: 1.759 loss_feat: 1.868 loss_dice: 0.4161 loss_rec: 3.952 loss_ce_0: 0.9591 loss_giou_0: 1.836 loss_bbox_0: 1.622 loss_feat_0: 1.862 loss_dice_0: 0.4251 loss_ce_1: 1.043 loss_giou_1: 1.668 loss_bbox_1: 1.536 loss_feat_1: 1.896 loss_dice_1: 0.4397 loss_ce_2: 0.9538 loss_giou_2: 1.663 loss_bbox_2: 1.605 loss_feat_2: 1.846 loss_dice_2: 0.4208 loss_ce_3: 0.955 loss_giou_3: 1.872 loss_bbox_3: 1.652 loss_feat_3: 1.909 loss_dice_3: 0.4367 loss_ce_4: 0.9662 loss_giou_4: 1.898 loss_bbox_4: 1.971 loss_feat_4: 1.873 loss_dice_4: 0.4096 time: 0.5228 data_time: 0.0024 lr: 2.151e-05 max_mem: 14227M [08/23 11:48:48 d2.utils.events]: eta: 0:01:01 iter: 879 total_loss: 53.8 loss_ce: 0.9774 loss_giou: 2.094 loss_bbox: 2.079 loss_feat: 1.949 loss_dice: 0.5008 loss_rec: 4.018 loss_ce_0: 1.006 loss_giou_0: 2.566 loss_bbox_0: 3.614 loss_feat_0: 1.95 loss_dice_0: 0.5281 loss_ce_1: 1.075 loss_giou_1: 1.889 loss_bbox_1: 2.011 loss_feat_1: 2.023 loss_dice_1: 0.5303 loss_ce_2: 1.059 loss_giou_2: 2.556 loss_bbox_2: 1.972 loss_feat_2: 1.957 loss_dice_2: 0.5211 loss_ce_3: 0.9959 loss_giou_3: 2.086 loss_bbox_3: 2.429 loss_feat_3: 1.918 loss_dice_3: 0.5039 loss_ce_4: 1.099 loss_giou_4: 2.365 loss_bbox_4: 2.467 loss_feat_4: 1.967 loss_dice_4: 0.5015 time: 0.5234 data_time: 0.0109 lr: 2.2005e-05 max_mem: 14227M [08/23 11:49:00 d2.utils.events]: eta: 0:00:51 iter: 899 total_loss: 55.53 loss_ce: 0.8718 loss_giou: 2.577 loss_bbox: 2.895 loss_feat: 1.908 loss_dice: 0.5386 loss_rec: 4.416 loss_ce_0: 0.9873 loss_giou_0: 1.9 loss_bbox_0: 2.658 loss_feat_0: 1.894 loss_dice_0: 0.5046 loss_ce_1: 0.9631 loss_giou_1: 2.026 loss_bbox_1: 2.844 loss_feat_1: 1.854 loss_dice_1: 0.49 loss_ce_2: 0.8662 loss_giou_2: 2.29 loss_bbox_2: 2.941 loss_feat_2: 1.912 loss_dice_2: 0.5419 loss_ce_3: 0.8571 loss_giou_3: 2.35 loss_bbox_3: 3.014 loss_feat_3: 1.88 loss_dice_3: 0.5226 loss_ce_4: 0.8432 loss_giou_4: 2.36 loss_bbox_4: 3.06 loss_feat_4: 1.897 loss_dice_4: 0.5402 time: 0.5240 data_time: 0.0023 lr: 2.25e-05 max_mem: 14227M [08/23 11:49:10 d2.utils.events]: eta: 0:00:41 iter: 919 total_loss: 49.52 loss_ce: 0.8145 loss_giou: 2.221 loss_bbox: 1.703 loss_feat: 1.819 loss_dice: 0.4973 loss_rec: 4.487 loss_ce_0: 0.9167 loss_giou_0: 1.974 loss_bbox_0: 1.7 loss_feat_0: 1.887 loss_dice_0: 0.4897 loss_ce_1: 0.95 loss_giou_1: 2.039 loss_bbox_1: 2 loss_feat_1: 1.88 loss_dice_1: 0.4829 loss_ce_2: 0.9038 loss_giou_2: 2.148 loss_bbox_2: 2.248 loss_feat_2: 1.921 loss_dice_2: 0.4869 loss_ce_3: 0.9378 loss_giou_3: 2.511 loss_bbox_3: 2.044 loss_feat_3: 1.873 loss_dice_3: 0.4899 loss_ce_4: 0.8742 loss_giou_4: 2.291 loss_bbox_4: 1.89 loss_feat_4: 1.862 loss_dice_4: 0.4915 time: 0.5242 data_time: 0.0023 lr: 2.2995e-05 max_mem: 14227M [08/23 11:49:21 d2.utils.events]: eta: 0:00:30 iter: 939 total_loss: 45.95 loss_ce: 0.9701 loss_giou: 2.326 loss_bbox: 1.83 loss_feat: 1.893 loss_dice: 0.434 loss_rec: 3.968 loss_ce_0: 1.002 loss_giou_0: 1.797 loss_bbox_0: 1.599 loss_feat_0: 1.892 loss_dice_0: 0.4256 loss_ce_1: 0.9709 loss_giou_1: 1.753 loss_bbox_1: 1.484 loss_feat_1: 1.957 loss_dice_1: 0.4503 loss_ce_2: 0.9275 loss_giou_2: 1.814 loss_bbox_2: 1.459 loss_feat_2: 1.9 loss_dice_2: 0.4727 loss_ce_3: 1.146 loss_giou_3: 1.825 loss_bbox_3: 1.41 loss_feat_3: 1.902 loss_dice_3: 0.455 loss_ce_4: 1.188 loss_giou_4: 2.023 loss_bbox_4: 1.541 loss_feat_4: 1.929 loss_dice_4: 0.4597 time: 0.5248 data_time: 0.0383 lr: 2.349e-05 max_mem: 14227M [08/23 11:49:31 d2.utils.events]: eta: 0:00:20 iter: 959 total_loss: 45.67 loss_ce: 0.8373 loss_giou: 2.285 loss_bbox: 1.503 loss_feat: 1.878 loss_dice: 0.4484 loss_rec: 4.1 loss_ce_0: 0.9539 loss_giou_0: 1.986 loss_bbox_0: 1.67 loss_feat_0: 1.98 loss_dice_0: 0.437 loss_ce_1: 0.9962 loss_giou_1: 1.8 loss_bbox_1: 1.673 loss_feat_1: 1.902 loss_dice_1: 0.4472 loss_ce_2: 1.143 loss_giou_2: 1.755 loss_bbox_2: 1.554 loss_feat_2: 2.013 loss_dice_2: 0.4528 loss_ce_3: 0.9458 loss_giou_3: 1.764 loss_bbox_3: 1.725 loss_feat_3: 1.865 loss_dice_3: 0.5278 loss_ce_4: 0.9717 loss_giou_4: 1.981 loss_bbox_4: 1.569 loss_feat_4: 1.895 loss_dice_4: 0.4281 time: 0.5243 data_time: 0.0022 lr: 2.3985e-05 max_mem: 14227M [08/23 11:49:42 d2.utils.events]: eta: 0:00:10 iter: 979 total_loss: 45.14 loss_ce: 0.9397 loss_giou: 2.134 loss_bbox: 1.692 loss_feat: 1.85 loss_dice: 0.4862 loss_rec: 3.837 loss_ce_0: 0.933 loss_giou_0: 1.952 loss_bbox_0: 1.451 loss_feat_0: 1.813 loss_dice_0: 0.461 loss_ce_1: 0.9746 loss_giou_1: 1.909 loss_bbox_1: 1.321 loss_feat_1: 1.862 loss_dice_1: 0.4729 loss_ce_2: 0.969 loss_giou_2: 2.166 loss_bbox_2: 1.522 loss_feat_2: 1.805 loss_dice_2: 0.4938 loss_ce_3: 0.8606 loss_giou_3: 2.204 loss_bbox_3: 2.024 loss_feat_3: 1.834 loss_dice_3: 0.4875 loss_ce_4: 0.8982 loss_giou_4: 2.013 loss_bbox_4: 1.619 loss_feat_4: 1.867 loss_dice_4: 0.4872 time: 0.5242 data_time: 0.0023 lr: 2.448e-05 max_mem: 14227M [08/23 11:49:52 fvcore.common.checkpoint]: Saving checkpoint to ./output/model_0000999.pth [08/23 11:49:56 fvcore.common.checkpoint]: Saving checkpoint to ./output/model_final.pth [08/23 11:50:01 d2.utils.events]: eta: 0:00:00 iter: 999 total_loss: 53.38 loss_ce: 0.8902 loss_giou: 3.072 loss_bbox: 2.707 loss_feat: 1.892 loss_dice: 0.4868 loss_rec: 3.579 loss_ce_0: 1.053 loss_giou_0: 1.811 loss_bbox_0: 1.909 loss_feat_0: 1.944 loss_dice_0: 0.4691 loss_ce_1: 1.035 loss_giou_1: 1.866 loss_bbox_1: 1.825 loss_feat_1: 1.917 loss_dice_1: 0.4915 loss_ce_2: 0.9807 loss_giou_2: 2.223 loss_bbox_2: 1.864 loss_feat_2: 1.896 loss_dice_2: 0.4711 loss_ce_3: 0.9004 loss_giou_3: 2.458 loss_bbox_3: 2.193 loss_feat_3: 1.893 loss_dice_3: 0.4881 loss_ce_4: 0.9202 loss_giou_4: 2.728 loss_bbox_4: 2.35 loss_feat_4: 1.951 loss_dice_4: 0.5161 time: 0.5239 data_time: 0.0314 lr: 2.4975e-05 max_mem: 14227M [08/23 11:50:01 d2.engine.hooks]: Overall training speed: 998 iterations in 0:08:42 (0.5239 s / it) [08/23 11:50:01 d2.engine.hooks]: Total training time: 0:08:53 (0:00:11 on hooks) [08/23 11:50:01 d2.data.datasets.coco]: Loaded 300 images in COCO format from datasets/totaltext/totaltext_test.json [08/23 11:50:01 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(1000, 1000), max_size=1824, sample_style='choice')] [08/23 11:50:01 d2.data.common]: Serializing 300 elements to byte tensors and concatenating them all ... [08/23 11:50:01 d2.data.common]: Serialized dataset takes 0.54 MiB [08/23 11:50:01 d2.evaluation.evaluator]: Start inference on 300 images /home/pirl/Desktop/OCR/SwinTextSpotter/detectron2/evaluation/text_evaluation.py:476: RuntimeWarning: invalid value encountered in divide ovr = inter_areas[i][order[1:]] / ((areas[i]) + areas[order[1:]] - inter_areas[i][order[1:]]) [08/23 11:50:08 d2.evaluation.evaluator]: Inference done 11/300. 0.4119 s / img. ETA=0:02:38 [08/23 11:50:13 d2.evaluation.evaluator]: Inference done 23/300. 0.3875 s / img. ETA=0:02:12 [08/23 11:50:18 d2.evaluation.evaluator]: Inference done 35/300. 0.3772 s / img. ETA=0:02:01 [08/23 11:50:24 d2.evaluation.evaluator]: Inference done 47/300. 0.3745 s / img. ETA=0:01:56 [08/23 11:50:29 d2.evaluation.evaluator]: Inference done 58/300. 0.3727 s / img. ETA=0:01:51 [08/23 11:50:34 d2.evaluation.evaluator]: Inference done 65/300. 0.3753 s / img. ETA=0:01:55 [08/23 11:50:39 d2.evaluation.evaluator]: Inference done 72/300. 0.3765 s / img. ETA=0:01:57 [08/23 11:50:44 d2.evaluation.evaluator]: Inference done 80/300. 0.3755 s / img. ETA=0:01:56 [08/23 11:50:49 d2.evaluation.evaluator]: Inference done 90/300. 0.3727 s / img. ETA=0:01:50 [08/23 11:50:54 d2.evaluation.evaluator]: Inference done 98/300. 0.3734 s / img. ETA=0:01:48 [08/23 11:50:59 d2.evaluation.evaluator]: Inference done 101/300. 0.3754 s / img. ETA=0:01:54 [08/23 11:51:05 d2.evaluation.evaluator]: Inference done 105/300. 0.3767 s / img. ETA=0:01:57 [08/23 11:51:10 d2.evaluation.evaluator]: Inference done 112/300. 0.3760 s / img. ETA=0:01:55 [08/23 11:51:15 d2.evaluation.evaluator]: Inference done 122/300. 0.3756 s / img. ETA=0:01:47 [08/23 11:51:22 d2.evaluation.evaluator]: Inference done 128/300. 0.3763 s / img. ETA=0:01:48 [08/23 11:51:28 d2.evaluation.evaluator]: Inference done 132/300. 0.3772 s / img. ETA=0:01:51 [08/23 11:51:34 d2.evaluation.evaluator]: Inference done 135/300. 0.3786 s / img. ETA=0:01:53 [08/23 11:51:39 d2.evaluation.evaluator]: Inference done 138/300. 0.3798 s / img. ETA=0:01:55 [08/23 11:51:44 d2.evaluation.evaluator]: Inference done 143/300. 0.3804 s / img. ETA=0:01:53 [08/23 11:51:49 d2.evaluation.evaluator]: Inference done 150/300. 0.3798 s / img. ETA=0:01:48 [08/23 11:51:55 d2.evaluation.evaluator]: Inference done 152/300. 0.3812 s / img. ETA=0:01:51 [08/23 11:52:03 d2.evaluation.evaluator]: Inference done 162/300. 0.3811 s / img. ETA=0:01:43 [08/23 11:52:08 d2.evaluation.evaluator]: Inference done 170/300. 0.3803 s / img. ETA=0:01:37 [08/23 11:52:13 d2.evaluation.evaluator]: Inference done 180/300. 0.3790 s / img. ETA=0:01:28 [08/23 11:52:18 d2.evaluation.evaluator]: Inference done 191/300. 0.3779 s / img. ETA=0:01:18 [08/23 11:52:24 d2.evaluation.evaluator]: Inference done 201/300. 0.3787 s / img. ETA=0:01:10 [08/23 11:52:30 d2.evaluation.evaluator]: Inference done 213/300. 0.3773 s / img. ETA=0:01:00 [08/23 11:52:35 d2.evaluation.evaluator]: Inference done 222/300. 0.3773 s / img. ETA=0:00:54 [08/23 11:52:40 d2.evaluation.evaluator]: Inference done 232/300. 0.3766 s / img. ETA=0:00:46 [08/23 11:52:46 d2.evaluation.evaluator]: Inference done 243/300. 0.3765 s / img. ETA=0:00:38 [08/23 11:52:51 d2.evaluation.evaluator]: Inference done 249/300. 0.3774 s / img. ETA=0:00:34 [08/23 11:52:56 d2.evaluation.evaluator]: Inference done 259/300. 0.3784 s / img. ETA=0:00:27 [08/23 11:53:02 d2.evaluation.evaluator]: Inference done 269/300. 0.3795 s / img. ETA=0:00:20 [08/23 11:53:07 d2.evaluation.evaluator]: Inference done 278/300. 0.3813 s / img. ETA=0:00:14 [08/23 11:53:12 d2.evaluation.evaluator]: Inference done 287/300. 0.3823 s / img. ETA=0:00:08 [08/23 11:53:17 d2.evaluation.evaluator]: Inference done 292/300. 0.3825 s / img. ETA=0:00:05 [08/23 11:53:21 d2.evaluation.evaluator]: Total inference time: 0:03:16.941381 (0.667598 s / img per device, on 1 devices) [08/23 11:53:21 d2.evaluation.evaluator]: Total inference pure compute time: 0:01:52 (0.382618 s / img per device, on 1 devices) [08/23 11:53:21 d2.evaluation.text_evaluation]: Saving results to ./output/inference/text_results.json Calculated! "E2E_RESULTS: precision: 0, recall: 0.0, hmean: 0" "DETECTION_ONLY_RESULTS: precision: 0, recall: 0.0, hmean: 0" Calculated! "E2E_RESULTS: precision: 0, recall: 0.0, hmean: 0" "DETECTION_ONLY_RESULTS: precision: 0, recall: 0.0, hmean: 0" [08/23 11:53:22 d2.engine.defaults]: Evaluation results for totaltext_test in csv format: [08/23 11:53:22 d2.evaluation.testing]: copypaste: Task: DETECTION_ONLY_RESULTS [08/23 11:53:22 d2.evaluation.testing]: copypaste: precision,recall,hmean [08/23 11:53:22 d2.evaluation.testing]: copypaste: 0.0000,0.0000,0.0000 [08/23 11:53:22 d2.evaluation.testing]: copypaste: Task: None-E2E_RESULTS [08/23 11:53:22 d2.evaluation.testing]: copypaste: precision,recall,hmean [08/23 11:53:22 d2.evaluation.testing]: copypaste: 0.0000,0.0000,0.0000 [08/23 11:53:22 d2.evaluation.testing]: copypaste: Task: Strong-E2E_RESULTS [08/23 11:53:22 d2.evaluation.testing]: copypaste: precision,recall,hmean [08/23 11:53:22 d2.evaluation.testing]: copypaste: 0.0000,0.0000,0.0000

[shshwtsrkr]

Hi, can I know which GPU are you using and how much VRAM does it have ?

I know the question is not directly related but it could really help.

[yub2nn]

@shshwtsrkr Hi, I using 'Integrated Matrox G200eW3 Graphics Controller' 16000MB.

[mxin262]

Hi, you can increase the batch size and train more iterations. Besides, you can train the model based on our pre-training model.