Using TochMetrics for code simplification

[Borda]

:hammer_and_wrench: Proposed Refactor

We have been developing TorchMetrics to be a general-purpose metric and some domain-specific use-cases. In many cases, we have an exact mapping to scikit-learn with verification/testing to the reference metric for its correctness. The TM includes functional as well as nn.Module versions and for the most/standard metrics, the only dependency is PyTorch (for the domain-specific metrics you need to install related extras).

Suggest a potential alternative/fix

WIth using TM metrics you may rely on the widely tested correctness (testing against good standard pycocotools in multiple OS environments and all PyTorch versions above v1.4), and later you can use nn.Module to leverage update and compute. Moreover, the recent release v0.8 allows re-using/sharing common compete similar as you leverage confusion matrix.

Overall if you are fine with it, we are happy to draft a PR with a suggested change to verify in place the impact. If you have any questions, happy to follow up with me or @aniketmaurya

What I have quickly checked, all this segmentation_models_pytorch.metrics can be replaced by TM

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[Borda]

Mind have look @@aniketmaurya ^^

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[aniketmaurya]

hi @qubvel, I would like to create a PR for integrating TorchMetrics. Please let me know if you would like to see the integration 😄

[qubvel]

Hi @aniketmaurya Let's try to intargate it, it could be a great feature!

[krishanr]

Hello everyone, I show how easy it is to integrate Torchmetrics with segmentation models PyTorch (SMP) here. Hardly any additional code needed.

I've also taken a close look at the metrics available with segmentation models PyTorch and have a few comments:

1. The accuracy formula is incorrect in the multiclass class (more than 2 classes). I've checked that it disagrees with scikit-learn when the micro/macro options are chosen.
2. Metrics are incorrect when using class weights. For example the IoU score of an image with itself gives 1/num_classes.
3. As the above comments shows, this library can use further testing.

The above points give some support for only using Torchmetrics which is rigorously tested, and easy to use as my above blog post shows.

Also see this post: https://krishanr.github.io/fastpages/pytorch/torchmetrics/segmentationmodels/2022/09/06/multiclass_segmentation_metrics.html, where I gave a close comparison between Torchmetrics and segmentation models PyTorch. One key difference is that Torchmetrics replace 0/0 errors by 0 whereas SMP replaces them by 1 (under default settings).

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.