Special Tokens in glove.twitter.27B

[atebbifakhr]

Hi,

Can you explain meaning of special tokens in glove.twitter.27B?

for example: repeat, allcaps, elong, ...

[ghost]

Well, there are over 1 million words in the vocabulary, including those. Whenever you do a web scrape, all sorts of crazy unicode and other words show up. For the most part, you would use these word vectors by checking terms that come up in your own application against the dictionary, so you can safely ignore most of the surprising combinations of symbols.

[atebbifakhr]

So there is no preprocessing before learning word vectors? for example replacing all user mentions with or all urls with .

[atebbifakhr]

So there is no preprocessing before learning word vectors? for example replacing all user mentions with <> or all urls with <>.