Lightning-AI
Supporting mask
🚀 Feature
In addition to ignore_index, metrics should also support masks, which are more robust.
I posted this reply under a closed issue https://github.com/Lightning-AI/metrics/issues/54#issuecomment-1275195071, but it might have been overlooked because it's closed. So just wanted to draw attention to this. Feel free to close this and reopen the original issue, if appropriate.
Hi @ZhaofengWu, Thanks for raising this issue. We can definitely agree that this would be an awesome feature to include in torchmetrics. However, it will not be an priority for us right now.
Actually, I just realized that not all classes have a ignore_index. Then, for example, how should we do token-level regression, where the sequence lengths in a batch are variable, when MeanSquaredError doesn't take either ignore_index or a mask?
And this was also kind of my initial point. It doesn't make sense to have ignore_index in many settings such as regression, while masks are more universal.