Text and Vision tokens different from CLIP

[MLRadfys]

Hi and thanks for all the work done in this repository!

I noticed that the implementation of the CLS token in the Vision transformer, as well as the tokens used in the text transformer are different in you're implementation.

As far as I understand, CLIP attaches a CLS embeddings token before patch embeddings are send through the transformer. In this repo, it seems like the mean is computed over all patch embeddings instead, meaning the CLS token has no learnable parameters.

In addition, CLIP uses a <BOT> and a <EOT> token which are combined with the token embeddings in the beginning and end, respectively. In you're implementation, the text transformer uses a single CLS token.

Iam trying to make use of the FILIP part and incorporate it into the openAI implementation of CLIP. Unfortunately Iam somewhat unsure about how to handle the text tokens in the fine-grained loss.

When comparing patch token embeddings to text token embeddings should I ignore both the <BOT> and the <EOT> tokens?

Thanks in advance,

kind regards,

M