Need new scoring method to count inferred labels

[AHarouni]

Describe the solution you'd like After running batch inference on your data (labeled and/or unlabeled ), It would be useful to find volumes where the model missed an organ / disease. Or over estimated / under estimated an organ or disease. For example:

• Data set is for liver tumor, where are the images without tumors? This is either the AI missed the tumor and it must be labeled or it is a true normal and the data scientiest want to exclude this normal images
• Where are the images with 3 kidneys ? or kidney pixel count > xx pixels

This request may need to count pixels as well as calculate the actual volume in mm^3

Describe alternatives you've considered I managed to adapt the the code from the sum scoring as

import logging
import numpy as np
import torch
from monai.transforms import LoadImage
from monailabel.interfaces.datastore import Datastore, DefaultLabelTag
from monailabel.interfaces.tasks.scoring import ScoringMethod
from monailabel.interfaces.app import MONAILabelApp
from monailabel.interfaces.utils.app import app_instance

logger = logging.getLogger(__name__)

class LabelCount(ScoringMethod):
    """
    Compute pixel count for each label
    """
    def __init__(self):
        super().__init__("Compute label count for an inference ")

    def info(self):
        instance: MONAILabelApp = app_instance()
        dataStore: Datastore = instance.datastore()
        status = dataStore.status()
        label_tag = list(status['label_tags'].keys())

        return {
            "description": self.description,
            "config":
                {"label_tag": label_tag
                 }
        }

    def __call__(self, request, datastore: Datastore):
        loader = LoadImage(image_only=True)
        tag = request.get("label_tag", "")
        if not tag:
            logger.error(" scoring error! Need to pass a label_tag")
            return {}

        result = {}
        for image_id in datastore.list_images():
            label_id: str = datastore.get_label_by_image_id(image_id, tag)
            if label_id:
                uri = datastore.get_label_uri(label_id, tag)
                # logger.info(f" ============{label_id=} ===={uri=}")
                label = loader(uri)
                if isinstance(label, torch.Tensor):
                    label = label.numpy()

                lbs,lbs_count=np.unique(label, return_counts=True)
                lbs_count_dict={}
                for i, lb in enumerate(lbs):
                    lbs_count_dict[str(int(lb))]=int(lbs_count[i])  # int conversion is needed to avoid json error
                logger.info(f"============{label_id=} ===={uri=} organs found for {image_id} are {lbs_count_dict} ")

                # datastore.update_image_info(image_id, lbs_count_dict)
                datastore.update_label_info(label_id, tag, {"label_count": lbs_count_dict})
                result[label_id] = {"label_count": lbs_count_dict}
        return result

Additional context This needs an active learning to sort by the most or least number of pixels for each label.

[SachidanandAlle]

if this is generic enough and useful for everyone, please add this here: https://github.com/Project-MONAI/MONAILabel/tree/main/monailabel/tasks/scoring

The one provided are only sample examples.. however user can bring his/her own scoring methods in their App (which can be more specific to their use-case)

[AHarouni]

That is what I have done. I added it to my own app. If this is voted up we can push it to the out of the box as in first, sum and random