Best workflow for iterative training

[tiagofmc]

So I've been playing with LightFM for a while to give movie recommendations to users (I'm not using the MovieLens dataset).

I've also read the LightFM documentation several times, but still have some "basic" questions of how to do some stuff.

1. Right now I just add an additional number of entries to the matrix that are users/movies that still don't exist, and whenever we have a new interaction (user, movie), we just update that index. Is there any better way to add an interaction of a new user (that isn't yet in the matrix) to the model?

2. Is there a method to predict the scores of items given certain interactions? So instead of using the method predict() with user ID, we just send a list of interactions with movies and we get the scores of the items for those interactions.

Thanks in advance.

[maciejkula]

The features you are looking for are cold fold-in. Unfortunately they are not supported at the moment, but I am thinking of adding them as they are incredibly useful in many settings. For the moment what you suggest in (1) is the best you can do.

[nlassaux]

@maciejkula First thank you for your work! Cold fold-in seem to be a very valuable feature for lightfm, do you already have anything in the pipeline?

[maciejkula]

Not at the moment. I am working on a possible v2 which will feature fold-in, but I don't expect for this to be available soon.

[nlassaux]

Understood, thank you for your answer 👍

[dakl]

Hello!

I have an idea on how to implement this that I wanted to check with you, if this is still on the table.

The idea I have (a poc implementation of) is to be able to create a new model from an old model, initialise it with random values for the item and user embeddings and biases (as ususal) and then copy the ones that we're learned in the previous model over to the new one.

This requires two new parameters on the model, item_feature_names and user_feature_names in order to copy over the values to the correct places in the new model.

Something along the lines of

    @staticmethod
    def from_model(old_model, item_feature_names, user_feature_names):
        old_model._check_initialized()
        new_model = LightFM(
            no_components=old_model.no_components,
            item_feature_names=item_feature_names,
            user_feature_names=user_feature_names
            )
        new_model._initialize(
            no_components=new_model.no_components,
            no_user_features=len(user_feature_names),
            no_item_features=len(item_feature_names))

        item_feature_idx = np.in1d(old_model.item_feature_names, new_model.item_feature_names)

        user_feature_idx = np.in1d(old_model.user_feature_names, new_model.user_feature_names)

        new_model.item_embeddings[item_feature_idx] = old_model.item_embeddings
        new_model.user_embeddings[user_feature_idx] = old_model.user_embeddings
        new_model.item_biases[item_feature_idx] = old_model.item_biases
        new_model.user_biases[user_feature_idx] = old_model.user_biases

        return new_model

Usage would be old_model = LightFM(..., item_feature_names=item_feature_names, user_feature_names=user_feature_names), train, save the model, and then later

new_model = LightFM.from_model(
    old_model,
    item_feature_names=new_item_feature_names,
    user_feature_names=new_user_feature_names)

I can write this up as a proper PR of course, but wanted your input first. I'm guessing it's relevant to copy over the momentums and gradients from the old model as well.

cheers

[dakl]

@maciejkula pinging here since it's and old thread and this might be missed otherwise :)

[ezietsman]

@dakl I'm trying to do this right now, I've basically done what you have there but when I start training it using fit_partial the accuracies effectively starts back at zero and I need a similar number of epochs than just training it from scratch. Is there some place I can have a look at how to do the cold fold-in correctly?

[chris-boson]

@dakl @ezietsman Did you get anywhere with this approach?

[dbalabka]

@maciejkula could you please describe the idea of "fold-in" algorithm for LightFM or provide link to some paper for inspiration? I would like to review the possibility to implement it for LightFM.

[SimonCW]

I’m closing this issue because it has been inactive for a long time. If you still encounter the problem, please open a new issue.

Thank you!

[dbalabka]

@SimonCW IMO iterative training is an important feature for LightFM model which can be solved using "fold-in" approach. It would be good to keep this ticket open unless iterative training will be solved.

[SimonCW]

@dbalabka , sorry, I didn't see your most recent comment. I'm happy to leave it open for now.