understanding log,matrix of yolonas

[akashAD98]

im getting only a log file in my weights folder, there is no any graphs coming, it will be very easy to understand this graphs

here this are not clear im getting map 0.5 is 88 recall 97 precision map 6 % ,

which things should we consider to check our model performance?

[dagshub[bot]]

Join the discussion on DagsHub!

[NatanBagrov]

Hello, @akashAD98,

Exploring the experiment directory, you'll find tfevents file (maybe multiple files, depending on the amount of times you resumed training) - this is TensorBoard file that contains graphs and additional information.

Launch TensorBoard and watch the graphs:

tensordboard --logdir=/path/to/checkpoints

Let me know if that helps!

Please, close the issue once resolved.

[akashAD98]

@NatanBagrov why percision is lower? while recall & map is higer?

also while doing testing , what are the arguments can I pass? i want to pass the threshold?

im not getting the documentions for the arguments. thanks

[NatanBagrov]

Hi, precision and recall is a tradeoff. What is better depends on your setting (task, type 2 error). Please provide more details regarding where exactly you want to pass more arguments, so I can assist

[akashAD98]

@NatanBagrov yes im agree with you, but here is something wrong with matrix, recall is 98%,map 50 is 88 & precision is 6%, this is logically incorrect.

i want to get confidence of each detected objec+ bbox,labels & also how should i pass threshold argument while doing predictions

import torch
device = 0 if torch.cuda.is_available() else "cpu"
input_video_path = "video/test.mp4"
output_video_path = "opvideo/detectionst.mp4"
#device=0

best_model.to(device).predict(input_video_path).save(output_video_path)         
                        

[akashAD98]

also, i trained the custom model,detection are good but many false predictions, as compared to yolov8 model, so how can we improve it & reduce false predictions? concept of background images is still applied here?

also tell me the parameters we can tune to achieve better results

[NatanBagrov]

Hello, @Louis-Dupont can provide more details regarding .predict(...)

[akashAD98]

also not able to connect with my tensorbord

this is my file inside checkpint/

command i used tensordboard --logdir=/path/to/checkpoints events.out.tfevents.1683693565.sc-Sys-Ak.498657

[Louis-Dupont]

Hi @akashAD98 , The simplest way to increase precision (at the expense of recall) is to increase the confidence threshold. Predictions with a smaller confidence than the threshold will be ignored, mainly reducing the number of False Positives, but this can also affect the number of True Positives.

You can also set the IoU threshold if you have many bboxes that overlap. These parameters can be set when calling model.predict() (see doc), feel free to play with it :)

images_predictions = model.predict(IMAGES, iou=0.5, conf=0.7)

If you are working on multiple classes, you can also compute the confusion matrix of all these classes and then chose a custom confidence threshold for each class based on your needs. Then, you can manually filter the predictions with a similar logic to the snippet below:

conf_threshold = {
    "<class_name1>": 0.3,
    "<class_name2>": 0.7,
    ...
}

images_predictions = model.predict(IMAGES, iou=0.5, conf=0.3)

for image_prediction in images_predictions:
    class_names = image_prediction.class_names
    labels = image_prediction.prediction.labels
    confidence = image_prediction.prediction.confidence
    bboxes = image_prediction.prediction.bboxes_xyxy

    for i, (label, conf, bbox) in enumerate(zip(labels, confidence, bboxes)):
        class_name = class_names[int(label)]
        if conf >= conf_threshold[class_name]:
            ... 

Concerning the tensorboard, log-dir should be the path to the folder that includes your experiments, and not the path of your experiment tfevents iteself.

[akashAD98]

@Louis-Dupont still tensorboard was not working, i given path of my directory, why directly you can't save results in graph.

tensordboard --logdir=

its showing command not found,i have checked my tensorflow & tensorboard installed correctly.

below command is working

python3 -m tensorboard.main --logdir=~/my/training/dir

[akashAD98]

@Louis-Dupont should i need to set threshold value for pre at the time of traiing start

[Louis-Dupont]

@akashAD98 You can technically set the threshold value in your validation metric by setting score_thres to the DetectionMetrics. This will not affect training directly but will impact the computation of your metric.

In your case, what you want is to change the predictions, not to change the definition of your metric, so you should set the confidence threshold in the predict function like in the examples I showed you images_predictions = model.predict(IMAGES, conf=0.3).

[rexxsd]

I have the same issue!!Precision is really low

[shbkukuk]

I have YOLOV5 object detection model which trained with using custom dataset. So, I am trying to get object detection model same data with using YOLO-NAS. But The YOLO-NAS model can not achieve as mush as performants as YOLOV5. I didn't change params(use default params). Furthermore the metrics are weird. I have same issue as above comments. I get %99 recall and %85 mAP but precision is %0.04. Why is happened ? Params: train_params = { 'silent_mode': False, "average_best_models": True, "warmup_mode": "linear_epoch_step", "warmup_initial_lr": 1e-6, "lr_warmup_epochs": 3, "initial_lr": 5e-4, "lr_mode": "cosine", "cosine_final_lr_ratio": 0.1, "optimizer": "Adam", "optimizer_params": {"weight_decay": 0.0001}, "zero_weight_decay_on_bias_and_bn": True, "ema": True, "ema_params": {"decay": 0.9, "decay_type": "threshold"}, "max_epochs": args['epoch'], "mixed_precision": True, "loss": PPYoloELoss( use_static_assigner=False, num_classes=len(yaml_params['names']), reg_max=16 ), "valid_metrics_list": [ DetectionMetrics_050( score_thres=0.1, top_k_predictions=300, num_cls=len(yaml_params['names']), normalize_targets=True, post_prediction_callback=PPYoloEPostPredictionCallback( score_threshold=0.01, nms_top_k=1000, max_predictions=300, nms_threshold=0.7 ) ) ], "metric_to_watch": '[email protected]' }

[bzisl]

Same for me. Anyone has been able to get the confusion matrix ?

[Louis-Dupont]

Recall and Precision are computed for a confidence of 0.5, while [email protected] is the area under the precision-recall curve for IoU threshold of 0.5. Said in other words, while Recall and Precision represent a single point of the curve, [email protected] represents an integral. From there it does mathematically impossible or contradictory; it's not because the point of Confidence=0.5 yields a bad precision that there won't be a segment of that curve that would yield a great precision.

The [email protected] is meant to measure the prediction over multiple score thresholds. What the measures you shared show is that YoloNAS predicts a lot of FP with a confidence score of 0.5 (Precision) but few FN (Recall), and it seems to do well overall with higher score thresholds ([email protected]).

Eventually when you use the model you should find your optimal score threshold to fit your precision/recall requirements and use it, it doesnt really matter if you set it to 0.1, 0.5 or 0.9, it is all use-case dependant.

[akashpat1098]

@shbkukuk How do you created all those graph for YOLO NAS . Can you guide me on that. I also want to plot various performance metrics graph for my own custom trained model.