AP is below 0.2, but F1 is above 0.8

[hannaSkyrim]

Hi, my team has tested with efficientdet-d0 as base model and fine-tuned for tomatoes detection. We've noticed that whenever we trained and re-trained, even with large dataset of tomato (2500+ images) and augmentation, the AP still remained below 0.2. However, when we checked the inference results, F1 score is relatively OK (above 0.8) and confusion matrix also give out many True positives and some False Negatives. May i know why is this happening ? Is this normal ?

[fsx950223]

Could you share your evaluation command?

[td43]

@hannaSkyrim I am trying to calculate the confusion matrix. Did you calculate IOU? could you explain to me the steps needed? I found a repo where the author cm-tf2 explains how to get the confusion matrix using test tfrecord, but I was trying to do the same with a .txt file with the path to each test image.

[hannaSkyrim]

@danielTobon43 Hi, i did some research on the confusion matrix, and the only reference i could get is from here : https://github.com/Sujith93/Tensorflow2_Custom_objectionDetection/tree/master i somehow modified automl's model_inspect.py (in function saved_model_inference()) to generate the predicted csv based on Sujith93's test_pred_with_csv_gen.ipynb. and after we get this csv generated, we must compare the predicted results with the csv generated from the test set. this comparison is based from Sujith93's confusion_martix_object_detection.py

[hannaSkyrim]

@fsx950223 Hi, this is my train_and_eval parameter:

!python3 main.py --mode=train_and_eval
--train_file_pattern=train.record
--val_file_pattern=validate.record
--model_name=efficientdet-d0
--model_dir=trained-model/tomato-d0-G-20
--ckpt=efficientdet-d0
--train_batch_size=10
--eval_batch_size=8
--eval_samples=300
--num_examples_per_epoch=1200 --num_epochs=20
--hparams=config.yaml

in config.yaml: num_classes: 1 label_map: {1: tomato} jitter_min: 0.8 jitter_max: 1.2 mixed_precision: true

[fsx950223]

@fsx950223 Hi, this is my train_and_eval parameter:

!python3 main.py --mode=train_and_eval --train_file_pattern=train.record --val_file_pattern=validate.record --model_name=efficientdet-d0 --model_dir=trained-model/tomato-d0-G-20 --ckpt=efficientdet-d0 --train_batch_size=10 --eval_batch_size=8 --eval_samples=300 --num_examples_per_epoch=1200 --num_epochs=20 --hparams=config.yaml

in config.yaml: num_classes: 1 label_map: {1: tomato} jitter_min: 0.8 jitter_max: 1.2 mixed_precision: true

https://github.com/google/automl/issues/1114#issuecomment-957426145

[td43]

Hi @hannaSkyrim thanks for sharing. It is almost the same algorithm that I found on github. I found it more comfortable to have the validation set in tfrecord format, but thanks anyway. Now, I got a better understanding on how is the pipeline for the calculation of the confusion matrix.