nroduit
Seg Object not displayed
Describe the bug. What happened?
Test data: example_seg_objects.zip
Included are two seg objects with different issues, but both are not displayed in Weasis.
├── seg_anatomy.dcm # Shape and resolution matches reference image but with empty Derivation Image Sequence
└── seg_dgm_main.dcm # Different shape and spacing
To reference the DICOM standard:
If the referenced images have a defined Frame of Reference, the Segmentation Instance shall have the same Frame of Reference and is not required to have the same spatial sampling or extent as the referenced images.
https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_A.51.html
Both seg objects load and display in Slicer.
I wish they would in Weasis as well.
What version of Weasis are you running?
4.6.1
On which system the problem occurs?
Windows
Relevant log output
Additional contextual elements
No response
Hi,
I work a lot with DICOM segmentations and maintain a python library (highdicom) for creating them. @Spenhouet created this issue after a discussion on our issue thread and his segmentations were created using our library.
Weasis is a great viewer and we are using it for a few projects in my research group now. It would be really helpful if the support for correctly displaying DICOM segmentations improved. I thought it might be useful if I summarised my current understanding of its limitations (knowing the Segmentation format inside out as I do), because there are a few related but different problems here as far as I can tell. I am not a Java developer and haven't tried to understand the code, but have deduced this behaviour by experimenting with segmentation files I create.
It seems that Weasis looks only at the items in the Per Frame Functional Groups Sequence -> Derivation Image Sequence -> Source Image Sequence in order to work out how to display the segmentation over the source images or image frames. While simple, there are a few issues with this approach.
- Presence of an item within the Derivation Image Sequence implies frame-to-frame alignment with the source image, but does not necessarily imply pixel-to-pixel alignment. The frame of the segmentation image can be a flipped/rotated/scaled version of the source image frame and still have the Derivation Image Sequence populated. There is in fact a further attribute, the Spatial Locations Preserved attribute, that allows a segmentation to clearly state whether or not spatial locations are preserved in the derivation of the segmentation image from the source image. Unfortunately it is a type 3 (optional) attribute so it cannot be relied upon. However, Weasis appears to assume a segmentation frame is pixel-for-pixel aligned with any source frame specified in the Derivation Image Sequence. This can lead to it rendering valid segmentations incorrently where this is not the case, even in the situation where the segmentation explicitly declares that it is not aligned by specifying Spatial Locations Preserved as "NO". A simple improvement would be checking this attribute and not rendering if it is "NO". A better solution would be to check the spatial metadata in the Plane Position Sequence, Plane Orientation Sequence and Pixel Measures Sequence of the Per Frame or Shared Functional Groups Sequences, and use this to decide whether/how to re-orientate / re-scale the frames before displaying them over the segmentations.
- There may not be any items in the Derivation Image Sequence. This can occur when the segmentation is performed on a resampled version of the source image(s) such that there is no longer a 1:1 correspondence between source frame and segmentation frame. With modern AI based segmentation tools, this is actually very common (almost ubiquitous) because models are typically trained on volumes with a standardized spacing and orientation. Therefore to deal with the general case you need to be able to deal with arbitrary affine spatial relationships between source image and segmentation image. While I'm sure this is quite complicated, Weasis already has MPR capabilities that I imagine could be leveraged towards this goal.
In summary, I would recommend more or less ignoring the Derivation Image Sequence (except perhaps in situations where there is no frame of reference coordinate system at all), and using the spatial metadata in the segmentation.
While I cannot contribute Java code, I'm happy to provide test files and/or clarify anything above. Thanks again for a great viewer!
Tagging also @mccle who has also observed this behaviour and helped me figure it out.
Thank you for this information. Currently, we are not able to reslice the segmentation in any direction. I have started to implement support for other references using the PlanePositionSequence (see this code), but without applying reslicing according to transformations. If you have a set of datasets covering different cases, it would be very helpful. In any case, fully handling all scenarios will likely require significant changes in Weasis.
