Random -nan loss after evaluation

[ekesdf]

I have already checked my dataset for bad annotations all fine my image has the size 2387 by 3484 pixel and the label themselves are somewhere around 400-500 in width and 200-300 in height

This is my configfile train .txt

My cmd I used

.darknet/darknet detector train cfg/voc.data cfg/test1/test1.cfg "backup/yolov3.weights" -gpus 0,1

The Output from the cmd lines

 CUDA-version: 11000 (11000), cuDNN: 8.1.1, GPU count: 2  
 OpenCV isn't used - data augmentation will be slow 
0,1
 Prepare additional network for mAP calculation...
 0 : compute_capability = 370, cudnn_half = 0, GPU: Tesla K80 
net.optimized_memory = 0 
mini_batch = 1, batch = 16, time_steps = 1, train = 0 

 (next mAP calculation at 2500 iterations) 
 Last accuracy [email protected] = 0.63 %, best = 4.75 % 
 2500: -nan, -nan avg loss, 0.026100 rate, 5.876493 seconds, 80000 images, 2.912841 hours left

 calculation mAP (mean average precision)...
 Detection layer: 30 - type = 28 
 Detection layer: 37 - type = 28 
3
 detections_count = 0, unique_truth_count = 44  
class_id = 0, name = Char, ap = 0.00%   	 (TP = 0, FP = 0) 

 for conf_thresh = 0.25, precision = -nan, recall = 0.00, F1-score = -nan 
 for conf_thresh = 0.25, TP = 0, FP = 0, FN = 44, average IoU = 0.00 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision ([email protected]) = 0.000000, or 0.00 % 
Total Detection Time: 0 Seconds

Set -points flag:
 `-points 101` for MS COCO 
 `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) 
 `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset

 mean_average_precision ([email protected]) = 0.000000 
Saving weights to alex_model_training/backup/test3/train_last.weights
Loaded: 0.000107 seconds
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 30 Avg (IOU: 0.000000), count: 6, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 37 Avg (IOU: 0.000000), count: 1, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
 total_bbox = 120027, rewritten_bbox = 0.000000 % 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 30 Avg (IOU: 0.000000), count: 2, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 37 Avg (IOU: 0.000000), count: 2, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
 total_bbox = 120163, rewritten_bbox = 0.000000 % 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 30 Avg (IOU: 0.000000), count: 3, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 37 Avg (IOU: 0.000000), count: 1, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
 total_bbox = 120030, rewritten_bbox = 0.000000 % 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 30 Avg (IOU: 0.000000), count: 5, class_loss = -nan, iou_loss = -nan, total_loss = -nan 
v3 (giou loss, Normalizer: (iou: 0.50, obj: 1.00, cls: 1.00) Region 37 Avg (IOU: 0.000000), count: 1, class_loss = -nan, iou_loss = -nan, total_loss = -nan