Inconsistent pairwise hinge loss values when using masked labels

[Sahandw]

Issue type Bug

I'm noticing inconsistent behavior in tfr.keras.losses.PairwiseHingeLoss when using labels with negative values (which are masked) or when using precise masks with default reduction. Expectation is to have identical loss results when masks (labels with negative values) are used, however, I'm noticing inconsistent values. I think the reason for this is that loss is normalized to the sequence length irrespective of masks.

What makes this even more inconsistent is that when a lambda_weight is set for the loss (for example: tfr.keras.losses.NDCGLambdaWeight()), the results will be consistent.

import tensorflow_ranking as tfr
import tensorflow as tf

loss = tfr.keras.losses.PairwiseHingeLoss()
loss_weighted = tfr.keras.losses.PairwiseHingeLoss(lambda_weight=tfr.keras.losses.NDCGLambdaWeight())

y_true = tf.random.uniform(shape=(2, 16))
y_pred = tf.random.uniform(shape=(2, 16))
y_true_masked = tf.concat(
    [y_true, tf.ones(shape=(2, 16)) * -1.0], axis=1
)
y_pred_masked = tf.concat([y_pred, tf.random.uniform(shape=(2, 16))], axis=1)


# Expectation: return same values
loss(y_true, y_pred).numpy(), loss(y_true_masked, y_pred_masked).numpy() # output: (8.201115, 4.1005573)

loss_weighted(y_true, y_pred).numpy(), loss_weighted(y_true_masked, y_pred_masked).numpy() # output: (0.9949929, 0.9949929)