[Question] Sweep thresholds, how do they work?

[dalbis]

:question: Question

Hi I see that the package does some automatic sweeps to detect the best thresholds on the validation set. Can you explain briefly the contents of these files?

sweep_ensemble_iou.json sweep_model_iou.json sweep_model_nms_fn.json sweep_model_score_thresh.json sweep_remove_small_boxes.json

Do you think one could use those thresholds for filtering boxes inferred on an unseen test set?

[mibaumgartner]

Dear @dalbis ,

those jsons are intended for debug purposes and just show what is happening in the background (i.e. which values were tried and which results were computed). The inferred hyperparameters will automatically be used for inference on unseen data.

Inference Pipeline:

1. Model Min Predictions Score <- removes low scoring predictions, sweep_model_score_thresh.json
2. Remove Small Boxes <- remove small boxes, usually less important, sweep_remove_small_boxes.json
3. Model NMS <- apply models nms function, there are currently only two options in nndet, sweep_model_iou.json and sweep_model_nms_fn.json
4. Ensemble TTA and Model Predictions <- sweep_ensemble_iou.json, usually based on WBC

Hope this helps.

Best, Michael

[dalbis]

Thanks! I was not sure a thresholding on the score was done at inference time because I saw boxes with very little scores... and the first point above would suggest that they should be removed already. But perhaps it depends on the configuration?

[mibaumgartner]

Yes, the scores can remain quite low and the threshold depends on the config. Usually the threshold is rather small and the ensembling of the models can reduce the final scores even more.