I made my own pytorch object detection with my own datasets but map is not increasing

[SlowMonk]

can you guys look through my pytorch code and figure out which part did I do wrong?

https://github.com/SlowMonk/single_od_pytorch

simple one object detection with pytorch

[B]Epoch[0] loss[0.4150061990533556] MAP:[0.0%] Train loss decreased (0.900000 --> 0.415006). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[1] loss[0.3658331222832203] MAP:[0.10714285714285714%] Train loss decreased (0.415006 --> 0.365833). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[2] loss[0.3304239908854167] MAP:[0.11904761904761904%] Train loss decreased (0.365833 --> 0.330424). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[3] loss[0.3068529662809202] MAP:[0.11111111111111109%] Train loss decreased (0.330424 --> 0.306853). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[4] loss[0.2905081467969077] MAP:[0.11111111111111109%] Train loss decreased (0.306853 --> 0.290508). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[5] loss[0.27660175262107733] MAP:[0.11111111111111109%] Train loss decreased (0.290508 --> 0.276602). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[6] loss[0.26729247156454594] MAP:[0.11904761904761904%] Train loss decreased (0.276602 --> 0.267292). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[7] loss[0.262052005755582] MAP:[0.11904761904761904%] Train loss decreased (0.267292 --> 0.262052). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[8] loss[0.2574362024546616] MAP:[0.11904761904761904%] Train loss decreased (0.262052 --> 0.257436). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[9] loss[0.25671960690191814] MAP:[0.11904761904761904%] Train loss decreased (0.257436 --> 0.256720). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[10] loss[0.2525056573477658] MAP:[0.11904761904761904%] Train loss decreased (0.256720 --> 0.252506). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[11] loss[0.2502458925092859] MAP:[0.11904761904761904%] Train loss decreased (0.252506 --> 0.250246). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[12] loss[0.24753405503764914] MAP:[0.11904761904761904%] Train loss decreased (0.250246 --> 0.247534). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[13] loss[0.2460326980229239] MAP:[0.11904761904761904%] Train loss decreased (0.247534 --> 0.246033). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[14] loss[0.24430175177043392] MAP:[0.11904761904761904%] Train loss decreased (0.246033 --> 0.244302). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[15] loss[0.24367255065590143] MAP:[0.11904761904761904%] Train loss decreased (0.244302 --> 0.243673). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[16] loss[0.24275059456459613] MAP:[0.11904761904761904%] Train loss decreased (0.243673 --> 0.242751). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth [A]Epoch[17] loss[0.2418948710851726] MAP:[0.11904761904761904%] Train loss decreased (0.242751 --> 0.241895). Saving model ...to /home/jake/PycharmProjects/balloon_detection/weights/best_fasterrcnn_resnet50_fpn.pth