MONAILabel
MONAILabel copied to clipboard
Project-MONAI
Feature request: Support multiple annotations on the same image instance
To create high-quality datasets, it is necessary to have the same image annotated multiple times by different people (e.g. double or triple segmentations). At the moment there is no interface for this in MONAILabel.
My team could also contribute to this, but I am not sure what is the best way to implement this feature. Certainly some changes need to be made to the datastore, filter functions and plugins. Also some sort of user management is then necessary.
Currently, for multiple annotations, we use a workaround with multiple instances of MONAILabel sharing the same app. Each instance is then used by a different annotator. Later, the data must then be merged in a cumbersome way.
I would prefer that this functionality be integrated into MONAILabel in a bundled way. I am relatively sure that other teams will need this feature as well (@LilianaCaldeira, @dtruhn, @kbressem).
Server provides api to save label across multiple tags.. is that not useful? and at the end it provides full json for all image, labels (multiple of them mapped to their tags).. that should fairly help any client to consume and info and act something on top of it..
Note that for assessing intra-user variability it would be useful if multiple segmentations for the same image for the same user could be stored.
Would you be available to chat sometime next week and see how we can address this? As @SachidanandAlle says, MONAI Label has the API to tag multiple annotations. Happy to organize a meeting.
Thanks at all for continuously progressing MONAILabel. I just want to confirm the assumption by @jlvahldiek that this would be a desirable feature for other groups as well: in my experience inter-user variability assessment is almost always demanded by reviewers in higher-ranking peer-reviewed journals.
For the record improved handling of multiple labels per image (inter and intra-user) would be useful for my group as well.
One downstream feature I could see might be providing more flexibility as to which set of labels to use while training, or to perform some sort of aggregate operation on images with multiple labels, because presumably you don't want to use conflicting annotations simultaneously as a ground truth.
Would be happy to join a meeting if that's being made.
Would you be available to chat sometime next week and see how we can address this? As @SachidanandAlle says, MONAI Label has the API to tag multiple annotations. Happy to organize a meeting.
We are very well available next week Monday or Friday. Thank you very much for organizing the meeting, @diazandr3s!
Superb! What about next week on Thursday at 17:00 CET?
@dtruhn, @adamaji please send me your email address to [email protected] so I can also add you to the meeting.
@diazandr3s and @SachidanandAlle, unfortunately I will most likely not be able to attend the meeting next Thursday, sorry.
But I started PR #799 incorporating my ideas on this topic. I kindly ask for your pre-review.
From @muratmaga
There isn't anything in MONAI Label to stop a person hit the train button and starting retraining a model which might be an issue for a shared server (if multiple people do it).
I agree with this. Multi-user support should also differentiate between only-inference users and expert/admin users that can trigger the training.
Just wanted to add some meeting notes. @jlvahldiek @keno @adamaji feel free to add/comment on this.
XNAT / MONAI Label integration is great. But the following features are thought for the Slicer + Local file archive configuration.
Features:
1/ A mechanism to manage multiple annotations for the same volume 2/ User authentication 3/ Allow expert user to modify/update annotations
- This should be integrated with the MONAI Label Reviewer tool
- This PR is aligned to this feature by allowing two modes to MONAI Label: comparative and competitive
4/ Decouple inference and training capabilities in MONAI Label Module
Inference-only user interface:
Add password as well:
@diazandr3s Thank you very much for the summary. If the expert user updates a segmentation, there should be the possibility to either overwrite the original segmentation or save the updated segmentation as separate file. This would allow later assessment of inter-rater agreement.
Rather than overwriting previous maual segmentation, it might be nicer to keep all the versions with a flag to inform which version should be used for a given purpose. This leave the flexibility to use say either only the latest version or all available versions.
it might be nicer to keep all the versions with a flag to inform which version should be used for a given purpose
Instead of just a flag, a list of tags or key/value pairs could be associated with each segmentation. It could be used for all kinds of filtering and selection or store any additional metadata.
These extra metadata could be stored externally in some kind of database or additional metadata files, but may be useful to store in the segmentaiton file. The .seg.nrrd file format can store a list of arbitrary key/value pairs for each segment (in SegmentN_Tags) but we could easily introduce a convention for storing metadata for the entire segmentation.
The server already supports storing multiple labels.. each label while saving can be tagged based on user choice..
These extra metadata could be stored externally in some kind of database or additional metadata files, but may be useful to store in the segmentaiton file. The .seg.nrrd file format can store a list of arbitrary key/value pairs for each segment (in SegmentN_Tags) but we could easily introduce a convention for storing metadata for the entire segmentation.
I think it would be a great idea to store the tags directly in the segmentation file. If two files exist, there is always the possibility to "loose" one file, e.g., by renaming or moving the segmentation without updating the metadata file.
As I mentioned above.. server supports saving one label for each tag.. and you can as many as tags you want.. its more on client side logic how to make use of this and support different versions..
First of all, many many thanks for the commitment and the great work on MONAILabel.
1/ A mechanism to manage multiple annotations for the same volume
As @SachidanandAlle has already noted, these options are already available. In #799 I have implemented a corresponding draft.
2/ User authentication
User authentication with name and password would be great. Also user roles (e.g. annotator, expert annotator, admin) would be useful.
3/ Allow expert user to modify/update annotations
I agree that editability of annotations for expert users can be implemented relatively easily within MONAILabelReviewer. Appropriate metadata could be stored both server-side and in the header of the .seg.nrrd data.
4/ Decouple inference and training capabilities in MONAI Label Module
The user who triggers the training should be allowed to select the annotations to be used via tag list.