Cann't overfitt SparseInst

[sdimantsd]

trafficstars

Hi, I am try to overfit one image using sparse_inst_r50_giam config. I changed the dataset to a custom dataset and only 3 labels (car, bus, truck). Those are the lines in the log (after one day of training):

[05/11 12:29:09 d2.utils.events]:  eta: 9 days, 2:03:30  iter: 119739  total_loss: 0.873  loss_box: 0.4872  loss_obj_pos: 0.0006603  loss_obj_neg: 0.003101  loss_cls: 0.04907  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.128e-07  loss_wh: 0.02689  time: 0.5578  data_time: 0.0603  lr: 0.00011079  max_mem: 4517M
[05/11 12:29:20 d2.utils.events]:  eta: 9 days, 2:06:10  iter: 119759  total_loss: 0.8729  loss_box: 0.4872  loss_obj_pos: 0.0006603  loss_obj_neg: 0.0031  loss_cls: 0.04907  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.165e-07  loss_wh: 0.02699  time: 0.5578  data_time: 0.0607  lr: 0.00011079  max_mem: 4517M
[05/11 12:29:31 d2.utils.events]:  eta: 9 days, 2:03:08  iter: 119779  total_loss: 0.8728  loss_box: 0.4871  loss_obj_pos: 0.0006602  loss_obj_neg: 0.003099  loss_cls: 0.04906  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.095e-07  loss_wh: 0.02677  time: 0.5578  data_time: 0.0594  lr: 0.00011079  max_mem: 4517M
[05/11 12:29:42 d2.utils.events]:  eta: 9 days, 2:05:47  iter: 119799  total_loss: 0.8728  loss_box: 0.4871  loss_obj_pos: 0.0006602  loss_obj_neg: 0.003099  loss_cls: 0.04906  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.129e-07  loss_wh: 0.02687  time: 0.5578  data_time: 0.0605  lr: 0.00011078  max_mem: 4517M
[05/11 12:29:54 d2.utils.events]:  eta: 9 days, 2:08:15  iter: 119819  total_loss: 0.8727  loss_box: 0.487  loss_obj_pos: 0.0006601  loss_obj_neg: 0.003098  loss_cls: 0.04905  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.164e-07  loss_wh: 0.02697  time: 0.5578  data_time: 0.0610  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:05 d2.utils.events]:  eta: 9 days, 2:02:34  iter: 119839  total_loss: 0.8727  loss_box: 0.487  loss_obj_pos: 0.00066  loss_obj_neg: 0.003097  loss_cls: 0.04905  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.093e-07  loss_wh: 0.02675  time: 0.5578  data_time: 0.0596  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:16 d2.utils.events]:  eta: 9 days, 1:58:52  iter: 119859  total_loss: 0.8726  loss_box: 0.4869  loss_obj_pos: 0.00066  loss_obj_neg: 0.003097  loss_cls: 0.04904  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.128e-07  loss_wh: 0.02685  time: 0.5578  data_time: 0.0615  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:27 d2.utils.events]:  eta: 9 days, 1:56:41  iter: 119879  total_loss: 0.8726  loss_box: 0.4869  loss_obj_pos: 0.0006598  loss_obj_neg: 0.003096  loss_cls: 0.04904  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.166e-07  loss_wh: 0.02696  time: 0.5578  data_time: 0.0604  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:38 d2.utils.events]:  eta: 9 days, 1:52:10  iter: 119899  total_loss: 0.8725  loss_box: 0.4869  loss_obj_pos: 0.0006598  loss_obj_neg: 0.003096  loss_cls: 0.04903  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.093e-07  loss_wh: 0.02674  time: 0.5578  data_time: 0.0613  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:49 d2.utils.events]:  eta: 9 days, 1:50:12  iter: 119919  total_loss: 0.8725  loss_box: 0.4868  loss_obj_pos: 0.0006597  loss_obj_neg: 0.003095  loss_cls: 0.04902  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.127e-07  loss_wh: 0.02684  time: 0.5578  data_time: 0.0609  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:00 d2.utils.events]:  eta: 9 days, 1:43:43  iter: 119939  total_loss: 0.8724  loss_box: 0.4868  loss_obj_pos: 0.0006596  loss_obj_neg: 0.003094  loss_cls: 0.04902  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.166e-07  loss_wh: 0.02694  time: 0.5578  data_time: 0.0605  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:12 d2.utils.events]:  eta: 9 days, 1:42:02  iter: 119959  total_loss: 0.8724  loss_box: 0.4867  loss_obj_pos: 0.0006595  loss_obj_neg: 0.003094  loss_cls: 0.04901  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.092e-07  loss_wh: 0.02672  time: 0.5578  data_time: 0.0610  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:23 d2.utils.events]:  eta: 9 days, 1:40:19  iter: 119979  total_loss: 0.8723  loss_box: 0.4867  loss_obj_pos: 0.0006595  loss_obj_neg: 0.003093  loss_cls: 0.04901  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.128e-07  loss_wh: 0.02682  time: 0.5578  data_time: 0.0623  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:34 fvcore.common.checkpoint]: Saving checkpoint to output/coco_yolomask/model_0119999.pth
[05/11 12:31:34 d2.utils.events]:  eta: 9 days, 1:27:01  iter: 119999  total_loss: 0.8723  loss_box: 0.4866  loss_obj_pos: 0.0006594  loss_obj_neg: 0.003092  loss_cls: 0.049  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.162e-07  loss_wh: 0.02693  time: 0.5578  data_time: 0.0613  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:45 d2.utils.events]:  eta: 9 days, 1:24:30  iter: 120019  total_loss: 0.8722  loss_box: 0.4866  loss_obj_pos: 0.0006593  loss_obj_neg: 0.003091  loss_cls: 0.049  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.089e-07  loss_wh: 0.02671  time: 0.5578  data_time: 0.0606  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:57 d2.utils.events]:  eta: 9 days, 1:18:55  iter: 120039  total_loss: 0.8721  loss_box: 0.4866  loss_obj_pos: 0.0006592  loss_obj_neg: 0.00309  loss_cls: 0.04899  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.122e-07  loss_wh: 0.02681  time: 0.5578  data_time: 0.0609  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:08 d2.utils.events]:  eta: 9 days, 1:22:10  iter: 120059  total_loss: 0.8721  loss_box: 0.4865  loss_obj_pos: 0.0006591  loss_obj_neg: 0.00309  loss_cls: 0.04899  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.161e-07  loss_wh: 0.02691  time: 0.5578  data_time: 0.0604  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:19 d2.utils.events]:  eta: 9 days, 1:21:07  iter: 120079  total_loss: 0.872  loss_box: 0.4865  loss_obj_pos: 0.0006591  loss_obj_neg: 0.003089  loss_cls: 0.04898  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.092e-07  loss_wh: 0.02669  time: 0.5578  data_time: 0.0622  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:30 d2.utils.events]:  eta: 9 days, 1:23:45  iter: 120099  total_loss: 0.872  loss_box: 0.4864  loss_obj_pos: 0.0006591  loss_obj_neg: 0.003089  loss_cls: 0.04898  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.121e-07  loss_wh: 0.02679  time: 0.5578  data_time: 0.0615  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:41 d2.utils.events]:  eta: 9 days, 1:20:48  iter: 120119  total_loss: 0.8719  loss_box: 0.4864  loss_obj_pos: 0.000659  loss_obj_neg: 0.003088  loss_cls: 0.04897  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.158e-07  loss_wh: 0.02689  time: 0.5578  data_time: 0.0607  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:53 d2.utils.events]:  eta: 9 days, 1:13:26  iter: 120139  total_loss: 0.8719  loss_box: 0.4863  loss_obj_pos: 0.000659  loss_obj_neg: 0.003088  loss_cls: 0.04896  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.09e-07  loss_wh: 0.02668  time: 0.5578  data_time: 0.0608  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:04 d2.utils.events]:  eta: 9 days, 1:08:34  iter: 120159  total_loss: 0.8718  loss_box: 0.4863  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003087  loss_cls: 0.04896  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.123e-07  loss_wh: 0.02678  time: 0.5578  data_time: 0.0610  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:15 d2.utils.events]:  eta: 9 days, 1:10:41  iter: 120179  total_loss: 0.8718  loss_box: 0.4863  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003087  loss_cls: 0.04896  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.162e-07  loss_wh: 0.02688  time: 0.5578  data_time: 0.0616  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:26 d2.utils.events]:  eta: 9 days, 1:20:52  iter: 120199  total_loss: 0.8717  loss_box: 0.4862  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003086  loss_cls: 0.04895  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.086e-07  loss_wh: 0.02666  time: 0.5578  data_time: 0.0604  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:37 d2.utils.events]:  eta: 9 days, 1:17:36  iter: 120219  total_loss: 0.8716  loss_box: 0.4862  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003085  loss_cls: 0.04894  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.121e-07  loss_wh: 0.02676  time: 0.5578  data_time: 0.0617  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:48 d2.utils.events]:  eta: 9 days, 1:07:31  iter: 120239  total_loss: 0.8716  loss_box: 0.4861  loss_obj_pos: 0.0006588  loss_obj_neg: 0.003085  loss_cls: 0.04894  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.154e-07  loss_wh: 0.02686  time: 0.5578  data_time: 0.0609  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:59 d2.utils.events]:  eta: 9 days, 1:11:03  iter: 120259  total_loss: 0.8715  loss_box: 0.4861  loss_obj_pos: 0.0006588  loss_obj_neg: 0.003084  loss_cls: 0.04893  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.086e-07  loss_wh: 0.02665  time: 0.5578  data_time: 0.0598  lr: 0.00011077  max_mem: 4517M

This is the image (after one day of trainin g):

Why the overfit did not works? (When I used Yolact the overfit works after a 4-5 hours)

Thanks

[lucasjinreal]

I don't think this is not fit. It actually learned something. You need carefully check your lr, gama, steps, and even change the optimizer suite for your tiny dataset.