Model overfitting when finetuned on smaller data

[diksham-flwls]

Hi I am trying to finetune the model on a smaller dataset of 450 samples of greyhead renders that look like this

My training loss seems to be converging but the validation loss is diverging. I am freezing everything but the final layers but otherwise using the exact same code as the repo. When I plot the euclidean distance of the FC layer outputs I find that the distribution is pretty overlapping for matched (green) and unmatched pairs (red).