Question regarding verify_negative_samples in BayesianPersonalizedRanking

[julioasotodv]

Hi,

I have a question regarding the argument verify_negative_samples in implicit.bpr.BayesianPersonalizedRanking, since I am unsure on how it works.

Imagine that we have an implicit dataset, but instead of having unary data (only purchases/non-purchases) we have count data, such as the number of plays in the Last.fm dataset.

When sampling triplets (user, item1, item2), and assuming that we always have feedback for item1 (this is, its record is non-zero in the csr matrix), I guess that whether this triplet is skipped or not during training depends on both the value of item2 and the argument verify_negative_samples. There are four possibilities here:

1. user has never interacted with item2 (hence, its record is 0 in the user-item matrix) and verify_negative_samples is True
2. user has never interacted with item2 (hence, its record is 0 in the user-item matrix) and verify_negative_samples is False
3. user has interacted with item2 (hence, its record is nonzero in the user-item matrix) and verify_negative_samples is True
4. user has interacted with item2 (hence, its record is nonzero in the user-item matrix) and verify_negative_samples is False

What is unclear to me is the real meaning of verify_negative_samples. By looking at the code (https://github.com/benfred/implicit/blob/master/implicit/cpu/bpr.pyx#L237), it looks like the triplet would only be skipped in scenario 3, but I would like to confirm whether that is the case or not.

If that is the case, the docstring for that argument is:

When sampling negative items, check if the randomly picked negative item has actually been liked by the user. This check increases the time needed to train but usually leads to better predictions

Has anyone seen an example where skipping triplets where we have known values for item1 and item2 (hence, some notion of preference if for instance item1 is consumed more than item2 by the user) leads to better results? I would argue that it is the other way around...

Thank you!

[benfred]

You are right - currently we don't skip items in BPR if verify_negative_samples is False (cases 2&4). Likewise we're only checking if the item is present in the sparse matrix (case 3). To handle case 1 (the item is present in the sparse matrix but the value is 0) you should call eliminate_zeros on the sparse matrix first.

It would be interesting to play around with only skipping values if the item1 preference is more than item2 preference - but I'm guessing that this won't help much in practice.

[julioasotodv]

@benfred thank you for the response!

Actually, I was more interested in making sure that preferences between two "liked" items but with different intensity (say, a user has played song1 20 times and song2 only 3, hence he prefers song1 > song2) were being modeled with verify_negative_samples=False. I would say that these kind of triplets actually add very valuable information during the training process. (In addition to the typical modeling between (nonzero, zero) pairs).

In fact, unless I am wrong: by looking at how the weights are updated during learning in https://github.com/benfred/implicit/blob/master/implicit/cpu/bpr.pyx#L253, I would say that these preferences where both items are nonzero are not being correctly learned with verify_negative_samples=False: the sign the logistic output z variable in the code should change depending on whether the liked item value (as named in the code) is higher or lower that the disliked one. So far, it is assumed that liked can only be larger than diskliked; but with verify_negative_samples=False this could not be the case. Therefore, the learning would be wrong.

Actually, the fix should be quite easy: would just be the matter of checking the sign for the subtraction between both sampled items, and adjust the logistic output z variable sign accodringly (this way, the gradient updates would be always correct).

What do you think? Shall I open a PR? I think that making sure that preferences between two nonzero items are correctly learned can be a major benefit.

Thank you!