Approach for Content Recommendation Usecase

[LaxmanSinghTomar]

Hello folks, thanks for the great package. I'm looking to build a Content Recommendation Engine. The task is to recommend content to the sales representatives based on their role, skills, skill level, ongoing deal status, etc. In the future, these recommendations would be served within a chatbot.

• I've decided to use Twin Tower Model for building the Retrieval Model (using positive interactions i.e. user clicked on the content). From my understanding, the negative interactions would automatically get sampled by the package during the computation of the scores matrix. Is it true? If yes, how many positive interactions per user are required for a good model?

• Then use the Ranking Model with MMR (using users' explicit feedback on the consumed content) in order to obtain the best content predictions. Initially, I'm looking to use an XGBoost as it works well on tabular datasets and due to explainability reasons in contrast to Neural Network. Any suggestions?

• Later, in order to explain these recommendations, I've decided to use a model explainability approach like SHAP to obtain feature importance and use sentence templates to fill in the values like: "Hey, I think you need to improve at <-feature-value-> <-feature-type->, check this out <-content-info->".

Few questions:

1. What do you think of the entire approach? Does it seem feasible?
2. Which concerns in terms of data collection, and model training should I adhere to?
3. For new userIDs/itemIDs, as it's not an e-commerce platform, using a large enough hashing function seems reasonable in the beginning, that can be replaced by items-interacted with (in case of users) in the future. What should be done for new itemIDs?
4. Initially when we don't have interaction data at all, how do we approach content recommendation? Should we use basic features and create user and content embeddings, and look to create UserKNN or ItemKNN-based retrieval? Thoughts on this?
5. What changes are to be made for every new user and content added? Index to be updated or entire re-training should be triggered?
6. How can we collect negative interactions when only a single recommendation is to be served within the chatbot? Should we rely on post recommendation feedback (for example, please rate it on a scale of 1-5) other than clicks and use that as negative interaction?

Your suggestions would be really helpful: @maciejkula @patrickorlando

[patrickorlando]

Welcome @LaxmanSinghTomar!

Your general approach is great. Recommendation use cases are all very unique and so it's hard to prescribe or validate an approach without trying it. A lot of the answers to your questions are it depends, but I'll try to provide some thoughts. It's best you break down the problem into stages, validating your assumptions.

Addressing a few of your points:

• how many positive interactions per user are required for a good model?

  It depends, both on the nature of your product and how you choose to encode your users. If you query tower contains an embedding per user, then it might require more examples per user in order to prevent overfitting. If many of your users have a handful of positive interactions, then you run the risk of the model remembering those items, rather than generalising. If you have rich user attributes, you might be able to get away with fewer positive interactions than if you have few/noisy attributes.

  If, instead of a user_id embedding, you encode a user based on their attributes, plus a sequence/bag of their recent positive interactions, then you limit this ability for the model to overfit to individual users. If you have sufficient user attributes, you may be able to include users with only a single example. In this case your hybrid recommender will be a mix of content-based and collaborative filtering. Check out this paper Deep Neural Networks for YouTube Recommendations as an example.

• XGBoost for ranking seems reasonable.

• I can't really comment on using SHAP scores to provide explainability to the end user. Sounds interesting.

• Initially when we don't have interaction data at all, how do we approach content recommendation? Should we use basic features and create user and content embeddings, and look to create UserKNN or ItemKNN-based retrieval? Thoughts on this?

  You may be able to get away with using the same model, if it's cold start performance with only user attributes performs well enough. Again, you'll need to test.

• What changes are to be made for every new user and content added?

  If re-training is not too expensive, it is the simpler option and I would go with that. Otherwise using hash embeddings might be the way to go. If you design your query to not rely on the user_id, then you'll have more leeway on your re-training schedule.

• How can we collect negative interactions when only a single recommendation is to be served within the chatbot? Should we rely on post recommendation feedback (for example, please rate it on a scale of 1-5) other than clicks and use that as negative interaction

  Since you know the user saw this recommendation and didn't click on it, you can use it as a negative sample. This is essential CTR prediction. You can also ask users for an explicit rating, which may perform better, or you may want to use both of these in a multi-task model. But be careful here, there are many pitfalls that tie into the question below.

• Which concerns in terms of data collection, and model training should I adhere to?

  Once you deploy a recommendation system into your product, you will be changing the way user's interact with you content. If the only data you are collecting for your CTR prediction is coming from the items you have recommended to the user, then the model will encounter a feedback loop. You recommend a single piece of content. The user clicks it. This may have been a sub-optimal recommendation, but since the user couldn't click on any other item, this reinforces that recommendations. Your entire system could fall into a local minima. You need to be collecting data from other channels in your product in a way that the user can tell you something about their preferences. They might be able to search for content for example. Additionally if you are using click through as a target, you need to collect explicit negatives, which is harder to do. You might need to track item impressions to know which items the user didn't click on.

Good luck! an if you can, please share your learnings.