[Feature Request] Support for Instance Level SR Qualitative Annotations

[igoroctaviano]

Request

One kind of annotation that is quite common in practice is used to describe a qualitative property of the image at the series or instance level. For example, this kind of annotation can be used to indicate whether a specific image slice contains some finding or not or describe the quality of the image. DICOM allows capturing such annotations in SR using TID 1500, specifically using its subordinate template TID 1501 rows 1, 10b and 11.

We would like to see support for instance-level image annotations implemented in OHIF Viewer. This would mean that in the situation where a TID 1500 instance is encountered in a study, and that instance includes TID 1501 content pattern as in example below ("Measurement group" container that contains one or more IMAGE content item, and CODE content items), the values of the CODE content items are shown as image annotations. Those annotations could be shown as key:value labels in the bottom of the image view, only for the instances specified. For the example below, this would show "Target Region: Neck". A tooltip can be used to show Code/CodingSchemeDesignator for the CodeValues shown.

<CONTAINER:(,,"Imaging Measurement Report")=CONTINUOUS>
  <has concept mod CODE:(,,"Language of Content Item and Descendants")=(en-US,RFC5646,"English (United States)")>
  <has obs context CODE:(,,"Observer Type")=(121006,DCM,"Person")>
  <has obs context PNAME:(,,"Person Observer Name")="Anonymous^Reader">
  <has concept mod CODE:(,,"Procedure reported")=(363679005,SCT,"Imaging procedure")>
  <contains CONTAINER:(,,"Image Library")=CONTINUOUS>
  <contains CONTAINER:(,,"Imaging Measurements")=CONTINUOUS>
    <contains CONTAINER:(,,"Measurement Group")=CONTINUOUS>
      <has obs context TEXT:(,,"Tracking Identifier")="Annotations group x">
      <has obs context UIDREF:(,,"Tracking Unique Identifier")="1.2.826.0.1.3680043.8.498.11346640510041906666146760516895890504">
      <contains CODE:(,,"Target Region")=(69536005,SCT,"Head")>
      <contains CODE:(,,"Target Region")=(45048000,SCT,"Neck")>
      <contains IMAGE:(,,"Source")=(CT image,)>

Why should we prioritize this feature?

This feature is needed to display the results of analysis for one of the IDC use cases this feature will be applicable for a variety of other tasks and datasets that contain or will contain qualitative annotations and are expected to be in IDC

CT series and corresponding annotations in DICOM SR https://www.dropbox.com/s/v1bz77e4nmqpa9n/Qualitative_annotation_sample.zip?dl=0

Example 1

Example 2

https://user-images.githubusercontent.com/13886933/236054411-540b42e8-07cc-4d23-a34e-6ee8f12ded8d.mp4

[fedorov]

The situation is that we already have data in IDC that has this kind of annotations, and it works great in v2. If we switch to v3 in IDC, we would lose this feature and it would be a regression. I would like to understand how difficult would be to add this to v3.

Example: https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.32722.99.99.62087908186665265759322018723889952421.

[james-hanks]

Added feature request for new UI component. IDC to add more use cases

[fedorov]

IDC to add more use cases

Can you please clarify how we can help?

[igoroctaviano]

@fedorov The idea here is to collect more examples and use cases e.g. when these qualitative annotations are most useful when they are displayed as a viewport overlay. With more examples, we might be able to expand/adapt the solution a little bit to allow better customization of what could be displayed as an overlay beyond qualitative annotations.

[james-hanks]

Use case: UI component as workflow solution https://www.loom.com/share/5606990bfe5c408094dfd2bf6a78abb8?t=124 using json

[sedghi]

another example

[fedorov]

I have to say there is an important distinction between how it is done in that other platform and how it was done in v2 (the approach I prefer).

In the former, the label contains just the value, without communicating the concept. In v2 we implemented "concept: value, which I think makes a lot more sense.

[james-hanks]

@dan-rukas to provide sketch of UI component

[wayfarer3130]

There is a fair bit of UI on the right click context menu to allow setting such values on existing measurements once a new annotation type is created. That UI allows setting the code values for named attributes - the currently supported two are finding and findingSite, but the code is generic in that it is easy to add targetRegion. For the UI, the work required is:

1. Add a new tool type - maybe ImageTypeTool - this shows the GUI element down at the bottom, or perhaps has some external way of rendering the annotation, eg in an overlay.
2. Add an adapter to save/restore the image type tool
3. Add OHIF adapters to save/restore the type
4. Configure menus and/or toolbars to set the annotation types

All of the commands to set values on these are defined in commands modules, so they can be called from a toolbar, a button, a context menu etc. The context menus are context sensitive, so that you can, for example, show a different context menu depending on what type of ImageTypeTool being clicked on, or the area of the screen, or the procedure or hanging protocol applied.

When creating the tool, these additional options should be considered: a. Apply to entire series b. Apply to range of images in series (can start with a, and then have ability to start on this image or end on this image) c. Unique based on label type (eg creating a new ImageTypeTool of the same group of codes will remove any other labels on that image within the same group). Unique should be within the range area. d. UI for the tool should allow configuration to be provided as part of registration

It may well be desirable to allow such labels to be created directly from context menus, rather than needing to click on toolbar button, create the annotation and then click on something else. The flow probably should be open the context menu, then select the fiducial (image type), and it directly appears, without ever getting into a ImageTypeTool mode.

[igoroctaviano]

There is a fair bit of UI on the right click context menu to allow setting such values on existing measurements once a new annotation type is created. That UI allows setting the code values for named attributes - the currently supported two are finding and findingSite, but the code is generic in that it is easy to add targetRegion. For the UI, the work required is:

1. Add a new tool type - maybe ImageTypeTool - this shows the GUI element down at the bottom, or perhaps has some external way of rendering the annotation, eg in an overlay.
2. Add an adapter to save/restore the image type tool
3. Add OHIF adapters to save/restore the type
4. Configure menus and/or toolbars to set the annotation types

All of the commands to set values on these are defined in commands modules, so they can be called from a toolbar, a button, a context menu etc. The context menus are context sensitive, so that you can, for example, show a different context menu depending on what type of ImageTypeTool being clicked on, or the area of the screen, or the procedure or hanging protocol applied.

When creating the tool, these additional options should be considered: a. Apply to entire series b. Apply to range of images in series (can start with a, and then have ability to start on this image or end on this image) c. Unique based on label type (eg creating a new ImageTypeTool of the same group of codes will remove any other labels on that image within the same group). Unique should be within the range area. d. UI for the tool should allow configuration to be provided as part of registration

It may well be desirable to allow such labels to be created directly from context menus, rather than needing to click on toolbar button, create the annotation and then click on something else. The flow probably should be open the context menu, then select the fiducial (image type), and it directly appears, without ever getting into a ImageTypeTool mode.

@wayfarer3130 thanks for the input! For IDC's purposes, I think we should break this down to keep things simple for the first iteration and create subsequent issues. The most important bit for now is the rendering of the image-level qualitative annotations as an overlay at the bottom of the viewport (your first point). So we need to implement the new tool type and display it as an overlay. That's pretty much what we have in v2.

Considering that the first point is what we are going to do, I'm not sure where to implement this, how we are going to enable or disable this feature (config? hanging protocol?), or how it might be customized (which tags to render?).

[igoroctaviano]

@dan-rukas @james-hanks considering my previous comment, do we have the design available to start implementing this?

[sedghi]

No it is not available, maybe in 3.8

[dan-rukas]

Here is the draft design to align with new viewport design updates in 3.8. I'm adding a few overview notes below, but a more detailed discussion can happen by Zoom.

A few notes:

• The primary color and design aligns with a consistent color of the viewport overlays.
• Colors or icon variations can be used. See below. For consistency and readability, color is used as a small circle before the label.
• Position in the viewport should be discussed further. I'm showing an example of labels aligning to a % of the viewport height. These should not overlap with the bottom left viewport text overlays.

[fedorov]

It is not clear if it was taken into account that those annotations sometime can have modifiers (subordinate attributes). You can see the discussion and examples of how this was implemented in v2 in this issue: https://github.com/OHIF/Viewers/issues/3108.

[james-hanks]

Per discussion 12/15:

• Hover state for contextual label E.g. in this screenshot label 'topographical modifier' is hover contextual and location 'center' is displayed in text
• Default max number of tags displayed with ability to config
• Allow color to be configurable (and turned off)
• Hide/display all

[fedorov]

@james-hanks are there any updates about this issue?

[fedorov]

@james-hanks @sedghi we are still interested to get this done! Any updates on finalizing the desired design?

[james-hanks]

@fedorov let us re-examine the design briefly to make sure it is up to date with our current standard, since it has been a while. Then we will share the design with your team

[fedorov]

Thank you James, this would be great!

For a really long time the issue of lack of support for overlapping segments in https://github.com/OHIF/Viewers/issues/3496 was the main blocker for us to phase out v2, but now that there is good progress in that issue, we might have resources in IDC to allocate to this one next!

[james-hanks]

@fedorov Could your team take another look at this design and let us know if you have any input? We will incorporate the feedback into the design update. It will probably be a few weeks after wrapping up some other items to return the updated design. Feel free to message me on slack about the resourcing timeline and we can figure out the details.

[fedorov]

James, thank you for the update!

Can you clarify what is the meaning of the colored tags? Is it proposed to give developer the option to use or not use the color? Can colors be made more prominent?

As a reminder, the original design (at the top of the issue) used tags color-filled. md.ai implementation also uses tags that are color filled. The proposed design with either no color at all, or very minimal color, might be difficult to quickly read. What are your thoughts on this?

Original design:

md.ai implementation of this feature:

[StratifAI-David]

Hi all,

I'm project manager in a very innovative research project called ODELIA (ODELIA – Open Source Swarm Learning to Empower Medical AI), where we use a technique called swarm learning to train a Breast Cancer AI model on data from different sites all over Europe, while at the same time preserving data privacy.

We intend to use your viewer for display of AI results, and this feature would be super-useful. So sorry to interrupt the discussion flow here, but in case there's a chance feature could be up-prioritized, this would be great. It would become part of something big :-)

Best, David

[dan-rukas]

@fedorov That makes sense. The design did significantly reduce color on the elements. I’ll explore a few options that bring back color that stands out while balancing readability.

We're currently finalizing our next release. Based on this feedback, I should have a few options that I can share for discussion here a few days after we ship.

@StratifAI-David No worries! This sounds great! Please stay subscribed to share feedback as well.

[StratifAI-David]

Hi Dan, thanks for your reply - yes for sure we'll share feedback, and thanks all for your great work!

[james-hanks]

@fedorov Based on the use cases you mentioned do you consider it a requirement to link the annotation label color with ROI or other annotation color? Or simply have the ability to customize the colors in the config?

[fedorov]

Based on the use cases you mentioned do you consider it a requirement to link the annotation label color with ROI or other annotation color? Or simply have the ability to customize the colors in the config?

It's a good question. I think this will depend on the nature of annotations and the specific use case. Our overall assumption is that qualitative annotations communicated via DICOM TID1500 SR instances. I can see the following scenarios:

1. Qualitative annotations could be associated with a DICOM image instance, where there is no geometric annotation and no other visualized component. In this case, the color would be assigned by the application, and could be customized in the config (as an example).
2. Annotations could be associated with contours or other geometric primitives, which do not have color assigned in DICOM, but which might have color assigned by the application (e.g., these annotations could be associated with RTSTRUCT structures, where color is not communicated in DICOM but is assigned by OHIF). In this case it would be beneficial to be able to synchronize the color between the annotation tag labels and structures colors.
3. Annotations could be associated with the regions defined by DICOM SEG segments, which may have color assigned in DICOM segment-level attributes. Again, would be good to sychronize.

I think overall, it is desirable to be able to control the color via some API and not just via config.