Better semantic segmentation plots

[adamjstewart]

Summary

There are various way we can improve upon our current semantic segmentation plots:

• [ ] Add legends to clarify what each class is
• [ ] Add button to control whether or not ground truth mask or model prediction is displayed (if both are included)
• [ ] Add slider to control alpha of mask

The latter two ideas bring TorchGeo plots closer to GIS plotting, making it easier for non-GIS experts to visualize their predictions.

Rationale

Datasets like CDL have 134 output classes. I have 2 or 3 of these memorized, but certainly not all 134.

Overlaying all plots on top of each other and allowing the user to control which mask is displayed and slide back and forth between image and mask will make it easier to explain model predictions.

The slider idea can also be applied to change detection plots, similar to https://aka.ms/libya_derna_flooding_2023

Implementation

Could do something similar to https://matplotlib.org/stable/gallery/widgets/slider_demo.html

Alternatives

No response

Additional information

No response

[calebrob6]

I think we have to be careful here as torchgeo isn't a viz library and questions like "how to make a legend support 134 classes" is in viz territory

[adamjstewart]

Of course. In that particular case, we could limit it to the 10 most popular classes or something like that. Just a suggestion for making the plots more useful.

[alhridoy]

Hi @adamjstewart

I completely agree that having legends would greatly enhance our understanding and visualization of each class.

I would like to work on this enhancement. Here's a brief plan:

1. Identify where and how the semantic segmentation mask plots are currently generated.
2. Design an intuitive and clear legend that can handle a large number of classes without overwhelming the visualization.
3. Implement the legend into the plotting function, ensuring it adapts to various numbers of classes.
4. Test with datasets of varying class numbers to ensure legibility and accuracy.
5. Update documentation or user guides, if needed, to explain the new feature.