Similarity scores do not look meaningful

[ekarabulut]

Hi

First of all, thank you for your work.

I am trying to run the trained model with some custom images given by me. The results did not make sense so I'd like to ask you if I'm missing something.

1- I prepared a small dataset of a few images dataset.zip

2- Then I created pairs.txt like this:

adile1.jpg adile2.jpg 1
adile1.jpg adile3.jpg 1 
sener1.jpg sener2.jpg 1 
munir2.jpg adile1.jpg 0 
munir1.jpg sener1.jpg 0
sener2.jpg adile2.jpg 0

3- I run evaluate.py in the LFW folder. I have obtained a similarity score for each pair which are given as follows:

adile1.jpg adile2.jpg 1 -4.43
adile1.jpg adile3.jpg 1 -2.06
sener1.jpg sener2.jpg 1 -0.07
munir2.jpg adile1.jpg 0 -0.88
munir1.jpg sener1.jpg 0 -3.38
sener2.jpg adile2.jpg 0 -5.03

The results confused me a bit. Similarity between "adile1" and "adile2" (same person) is -4.43 while similarity between "sener2" and "adile2" is -5.03. If you inspect the above list, it is hard to observe a proportion in the similarity score for similar and/or different people.

How can it be?

Secondly, is there a rough threshold for the similarity score between two images of the same person? For instance, score is between -5 and 0 for the same person and it is far less than this (e.g., -100) for different people?

Thanks in advance.

Best regards

[huangyangyu]

I found the problem. The images you suppled to me are not aligned and cropped. Please refer to util.py Moreover, you would better to normalize the feature vector. The effect may be better.

[ekarabulut]

@huangyangyu Thank you for your answer.

I understand each image should be aligned and cropped. I also assume this can be done by using functions given in util.py. Right?

I inspect the code (evaluate.py and featurer.py) and see that each image is passed through util.align_box function which presumably does the alignment job.

Therefore, I don't understand how further I can use util.py. It is already used in your code.

Do you refer to any other preprocesses other than things done in util.py? I'd be glad if you can clarify further.

EDIT: I checked the code one more time. This time, I saw the second parameter of util.align_box which is expected to have landmarks of the face features (eye_left, eye_right, nose etc) is always None. Therefore, util.align_box simply returns what is given (e.g. does nothing). I guess a python function that calculates landmarks of the detected face in the image is missing. Perhaps MTCNN is connected with the issue. It is possible I am getting it wrong. Any help is appreciated.

[huangyangyu]

@ekarabulut Yes. In our test of LFW and YTF, all the images are aligned by mtcnn and util.align_box. If you want to use evaluate.py script to test your own dataset, you should provide keypoints as parameter to util.align_box. You can get the keypoints by mtcnn or other landmark methods.

[ekarabulut]

Hi @huangyangyu

Thank you again for your feedback. I have recently had the opportunity to apply MTCNN (python implementation) to preprocess the images before giving to the SeqFace. It seems obtaining face landmark features and giving it to SeqFace works!

The new results with properly aligned face photos seem more meaningful.

• Faces of the same person give a cos similarity between -5 and -10
• Faces of different people give a cos similarity under -25

Therefore, I understand by applying a threshold, say -10, SeqFace is able to tell whether faces belong to different people or the same person. But I am not sure about how to pick a proper threshold for all cases as the value -10 is obtained as a result of my own experiments.

Can you tell me if my assumption is correct?

Thanks so far.

[isgursoy]

I think we can close this issue.