problem when swapped sentences

[ZeinabAghahadi]

I load the model and try to predict the similarity between sentence A and sentence B. when I change the order of these sentences (i change the place of sentence A and B and swapped them), i get different prediction value for similarity. Why this happen?is not this measure symmetric?

[AndriyMulyar]

This happens because switching the sentence order will influence which pre-trained attention weights each token enters. In theory, (semantic) similarity should be symmetric - the way BERT works makes this nearly impossible.

I tried to augment the number of training instances by exploiting this symmetry. In practice, it did not lead to any real improvement in generalization.

[ZeinabAghahadi]

OK. I understood what you mean. Thank you so much

[yuchenlin]

maybe SentenceBERT can help in this case?