Sequential model improvements

[pshashk]

Hi! Very cool project.

There are some potential improvements to sequential model found in Improved Recurrent Neural Networks for Session-based Recommendations.

Randomly dropping items from sequences helps to avoid over-fitting for website structure and improves MRR by ~8% in my experiments with proprietary click-stream data. If done per-batch at training phase the memory overhead could be avoided.

Faster recurrent units like GRU, QRNN or SRU could strike a better performance/accuracy trade-off than causal convolution model.

[maciejkula]

Thanks!

Sequence-based dropout looks like a good idea, I'd be very happy to add it (as well as GRUs and other types of representations).

Would you be willing to collaborate on this?

[pshashk]

Thanks for the offer. Unfortunately, right now I don't have enough spare time, but I will be glad to collaborate in the future.

[maciejkula]

That would be great when you have time. All of these look like worthwhile additions.

[halflings]

This is definitely worth checking for these models: https://github.com/hidasib/GRU4Rec

[elanmart]

Taking ideas from Deep Learning for NLP, things I believe would be nice and easy to try out on LSTM model:

• [x] self-attention on the sequence of items
• [ ] recurrent dropout,
• [ ] dense / residual connections

I'll try to play around with this during the weekend.