Lower recall and precision than original version

[hoonkai]

I've been running this on the KITTI dataset on and off for a while now, but I still cannot produce the results I can produce using the original TF SqueezeDet. I can achieve around 85% for car, 75% for pedestrian using the original, but I can only achieve around 80% for car, 55% for pedestrian using this version after 100 epochs. mAP doesn't get much higher than 60%. Has anyone tried training for 100 epochs and care to share their results? Anyone know why the original version gives better performance?

[Sileadim]

Hey. This could have several reasons:

• The original had some flag that excluded hard examples.
• It used a C++ script for evaluation. Maybe there is something different in there. Or we introduced a bug.
• It used a decay of learning rate after 10000 steps. We replaced this with the ReduceLROnPlateau Callback.
• Other hyperparameters might have changed

I did a training and after 260 epochs it reached a recall on cars of 83,4%. So maybe it has to do with the learning rate.

[hoonkai]

Hi

The original had some flag that excluded hard examples.

This is true, but the precision I'm getting is substantially lower than the precision I get for hard examples using the TF version. For instance, I can get around 70% for hard cyclist with the TF version, but less than 60% with this Keras version. Similarly, around 65% for hard pedestrian with TF but around 10% less with this Keras version.

I did a training and after 260 epochs it reached a recall on cars of 83,4%. So maybe it has to do with the learning rate.

My recall is generally quite good too, but it's the precision (and mAP) that's a fair bit off. Could you also share the precision values you achieved for cars, cyclist and pedestrian?

[Sileadim]

With a confidence threshold of 0: Class cyclist Precision: 0.11997670355270122 Recall: 0.6477987421381611 AP: 0.5933366847052632 Class pedestrian Precision: 0.13854722673473802 Recall: 0.6589403973509206 AP: 0.6208209806441031 Class car Precision: 0.2739594510645492 Recall: 0.8337916809900165 AP: 0.8394447285561287

With a confidence of .5: Class cyclist Precision: 0.619528619528411 Recall: 0.5786163522010759 AP: 0.8176477059944383 Class pedestrian Precision: 0.5334572490705823 Recall: 0.6232356134635588 AP: 0.7813052596437094 Class car Precision: 0.7592973571767565 Recall: 0.8238324175824034 AP: 0.920408962720865

With a confidence of .6:

Class cyclist Precision: 0.7333333333329841 Recall: 0.48427672955959616 AP: 0.879332719095202 Class pedestrian Precision: 0.6395631067960389 Recall: 0.5648445873525654 AP: 0.8298965908426681 Class car Precision: 0.821552924791072 Recall: 0.8098506950403156 AP: 0.9313479999167555

[IgorMunizS]

@Sileadim Could you please tell me what confidence threshold means? It's related with cfg.FINAL_THRESHOLD, rgiht?

Is there any method to eval AP in easy, moderate and hard mode on KITTI dataset?

Thanks!