Fine tuning help

[pv4n]

I tried standard embeddings but the model is not discerning between brothers well. So I decided to try finetuning casia-webface/vggface2 and classifying with 4 classes and 1000 images each. I ran the fine tuning script and got .0001 losses for both train and val but when I test the model in production, I get strange performance as the returned tensor is usually argmax at index 3 tensor([[-111.6994, -166.2166, 496.6208, -121.9813]]). When I train only the logits, the values are smaller tensor([[ 1.9653, -2.2871, 6.6909, -6.1794]]), but still falsely predicting class 3. Any clue what I may be doing wrong?

[Yassin-H-Rassul]

I have the same problem, I have trained on 5 classes and 15 images each. it always falsely predicts class 2.

I tried standard embeddings but the model is not discerning between brothers well. So I decided to try finetuning casia-webface/vggface2 and classifying with 4 classes and 1000 images each. I ran the fine tuning script and got .0001 losses for both train and val but when I test the model in production, I get strange performance as the returned tensor is usually argmax at index 3 tensor([[-111.6994, -166.2166, 496.6208, -121.9813]]). When I train only the logits, the values are smaller tensor([[ 1.9653, -2.2871, 6.6909, -6.1794]]), but still falsely predicting class 3. Any clue what I may be doing wrong?