timing benchmarks

[sandeepsoni]

I see in the README that there was some analysis done or planned about the timing of this code. I was wondering if a comparison with gensim's timing was ever made? I've been trying to get a working pytorch based implementation but still can't get it faster than gensim. I came across this repo which seems to be using tricks similar to what gensim did, so I'm curious.

[ray1007]

No I haven't compared it against gensim. I just realized I've left this repo for too long and the tables are still missing (LOL), I guess I should get some time to update this whole repo :P.

And yes the tricks are similar because I was inspired by these blog posts of gensim's author. part1 part2 part3.

And I think a pytorch implementation might at its best be only as good as gensim. I did it in pytorch to have a flexible framework to try different models for embeddings and take advantage of the autograd feature. On the other hand gensim is totally optimized for word2vec model. IMO pytorch implementation being slower than gensim is the tradeoff for flexibility.

Some random thoughts: You might want to give it a try on precomputed sigmoid table. I guess one could by extending torch.autograd.Function and define your own forward and backward method.

[sandeepsoni]

I agree with the flexibility that pytorch offers but I'm still surprised that it takes way more time than gensim to learn the embeddings. I did a basic comparison between your implementation of sgns and gensim, and one epoch in gensim is 7x times faster (around 3mins) than pytorch (around 21 mins) with 4 workers in parallel for both. My own version of the code is similarly slower to gensim.

The pre-computed sigmoid table is a good idea. Thanks for the suggestion! It seems to me that the biggest improvement for gensim was by writing part of their code in cython and using BLAS trickery. I'm guessing that's where I should explore more.