Precision-Recall Curve

[mrgransky]

trafficstars

Hi,

I would like to create a precision recall curve for this model and dataset but I am confused how to create true_positive, false_negative, false_positive and true_negative.

Is this a correct approach:

tp, fp, fn, tn = 0,0,0,0
for i in range(3450):
   for j in range(neighbors):
       if(query_lbl[i] >= top_5[j]-2 and query_lbl[i] <= top_5[j]+2):
	   tp += 1
       else:
	   fp += 1

top_5 corresponds to the top 5 predicted places from reference database for a given query!

The confusion comes from the fact written in the article:

It is important to note that we consider a match is correct when the closest feature vector corresponds to a place within a 5 -frame window.

Does this statement mean a match is considered true_positive if a query is predicted within a 5-frame window of the reference database? Or true_positive is still assessed based on exact label from query to reference as follows:

Query_Season ---> Reference_Season
frame_i      ---> frame_i

for example:

Query = 0: Top 5 predicted places: [ 7  5 12 11 10] --> tp = 0, fp = 5, fn = ?, tn = ?
Query = 1: Top 5 predicted places: [ 7  8  6 10  4] --> tp = 0, fp = 5, fn = ?, tn = ?
Query = 2: Top 5 predicted places: [ 8  7 12 11 10] --> tp = 0, fp = 5, fn = ?, tn = ?
Query = 3: Top 5 predicted places: [ 8  7  6  4 12] --> tp = 1, fp = 4, fn = ?, tn = ?
Query = 4: Top 5 predicted places: [ 4 12  8  7  6] --> tp = 2, fp = 3, fn = ?, tn = ?

I don't know how to get fn and tn in the above code!

Cheers,

[valavanisleonidas]

probably a bit late but I think I have the answer. Many papers have a threshold to define a positive image (for example if the ratio of the distance of the closest and second closest is above a threshold then it is considered a positive). To be a true positive it has to be in that 5 frame window. Otherwise it is false positive.

For negative images, there are no false negatives since the correct retrieved image exists in the db. So if the ratio is not above the threshold then it is a false negative.

I am also working on this. Have you figured it out ?

[mrgransky]

Do you have some code snippet to exactly explain what you mean?

[valavanisleonidas]

lets say that for a query the top 2 distances of the retrieved images are dist1 and dist2 . The ratio is defined as

ratio = dist1 / dist2

then for this ratio and a threshold you have tp, tn and fn as follows

if ratio < thresh:
    if closest_image_retrieved in 5frames:
        tp += 1
    else:
        fp += 1
else:
    fn += 1

what I dont get is from this how do you calculate the precision recall curves from this?? calculating the recall values from this does not correspond to 0-1 at all 11 points.

[jmfacil]

Hi,

Small edit on @valavanisleonidas comment.

if ratio < thresh:
    if closest_image_retrieved in window5frames:
        tp += 1
    else:
        fp += 1
else:
    if closest_image_retrieved in window5frames:
        fn += 1
    else:
        tn += 1

otherwise fn could have higher value and therefore the recall will be lower. Also by having a threshold you could consider more neighbors and not only the closest.