How to work with non-ranking metrics?

[cBournhonesque]

Hi, How can I compute non-ranking metrics or univariate metrics, such as AUC, or recall? (i.e. computing metrics after unfurling the list of examples, similar to what is done in the univariate scorer).

If I understand correctly, the metrics are being updated here: https://github.com/keras-team/keras/blob/v2.7.0/keras/engine/training.py#L817

The shapes of y and y_pred will be (batch_size, list_size). However, I don't have the mask anymore to know where the lists end, so I cannot 'unflatten' the lists myself.

Is there an example on how to do this? Maybe using a callback? Or overriding the train_step method so that I can pass the x variable inside the update_state function?

[cBournhonesque]

As a side note: do ranking metrics work with the current masking simply because those metrics are not affected by makes values where y_pred = ln(1e-10), y_true=0 ? Wouldn't it be more efficient to apply a boolean mask to remove the mask values entirely before passing the y_pred and y_true to the loss?

(same thing for ranking metrics, which don't seem to be affected by masked values. However, some non-ranking metrics like Accuracy or AUC will be affected)