[Bug][PlanarFreehandContourSegmentationTool] A drawn SegmentationRepresentations.Contour on one view does not show across multiple orthographic viewports

[prerakmody]

Describe the Bug

Task

• When using ViewportType.ORTHOGRAPHIC, making contours with PlanarFreehandContourSegmentationTool and using SegmentationRepresentations.Contour, I am unable to see the contours reflected across different views (i.e. axial, sagittal, coronal). Check this netlify example1.
  • Use the ReferenceLines (in green) to align the views together
  • Draw a contour on any one view (e.g. axial). It will NOT show up in the other view.

Alternative

• However, when using the same viewports but drawing with BrushTool and using SegmentationRepresentations.Labelmap I am able to see the labelmap across different contours. Check this other netlify example2.
  • Use the ReferenceLines (in green) to align the views together
  • Draw a labelmap using brush on any one view (e.g. axial). It WILL show up in the other view.
  • This behaviour is similar to the image in this documentation

Versions

"@cornerstonejs/core": "^1.78.1",
"@cornerstonejs/tools": "^1.77.13",

Why is this important

• Many commercial contouring tools, use contours during the segmentation phase (check links here). Hence, clinicians get used to the smoothness of these contours compared to the "jaggedness" of labelmap contours. This is further exacerbated when the 3D volume spacing is low.

Steps to Reproduce

Full code

• "Inspect" the netlify webpage --> go to "Sources" tab and refer to the .js file under visualiser/src
  • index3DContours.js (which uses contours - example1)
  • index3DBrush.js (which uses labelmap - example2)

Basic code for both examples

(not in order)

const viewportInputArray = [
        {viewportId: viewportIds[0],type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC,element: element1,defaultOptions: {orientation: cornerstone3D.Enums.OrientationAxis.AXIAL,},},
        {viewportId: viewportIds[1],type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC,element: element2,defaultOptions: {orientation: cornerstone3D.Enums.OrientationAxis.SAGITTAL,},},
        {viewportId: viewportIds[2],type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC,element: element3,defaultOptions: {orientation: cornerstone3D.Enums.OrientationAxis.CORONAL,},},
    ];

const renderingEngine = new cornerstone3D.RenderingEngine(renderingEngineId);
  
renderingEngine.setViewports(viewportInputArray);

await cornerstone3D.setVolumesForViewports(renderingEngine, [{ volumeId }], viewportIds); 

renderingEngine.renderViewports(viewportIds);

const segmentationDisplayTool = cornerstone3DTools.SegmentationDisplayTool;
const toolGroup = cornerstone3DTools.ToolGroupManager.createToolGroup(toolGroupId);
toolGroup.addTool(segmentationDisplayTool.toolName);
toolGroup.setToolEnabled(segmentationDisplayTool.toolName);

await cornerstone3D.volumeLoader.createAndCacheDerivedSegmentationVolume(volumeId, {volumeId: segmentationId,});

Example 1 - with contours

const planarFreehandContourSegmentationTool = cornerstone3DTools.PlanarFreehandContourSegmentationTool;
toolGroup.setToolActive(planarFreehandContourSegmentationTool.toolName, {bindings: [{mouseButton: cornerstone3DTools.Enums.MouseBindings.Primary, },],});// Left Click

segmentation.addSegmentations([ { segmentationId, representation: { type: cornerstone3DTools.Enums.SegmentationRepresentations.Contour, data: { volumeId: segmentationId, },}, }, ]);
const segmentationRepresentationUIDs = await segmentation.addSegmentationRepresentations(toolGroupId, [ { segmentationId, type: cornerstone3DTools.Enums.SegmentationRepresentations.Contour, }, ]);
segmentation.activeSegmentation.setActiveSegmentationRepresentation(toolGroupId,segmentationRepresentationUIDs[0]);

Example 2 - with labelmap

const brushTool  = cornerstone3DTools.BrushTool;
toolGroup.addToolInstance(strBrushCircle, brushTool.toolName, { activeStrategy: 'FILL_INSIDE_CIRCLE', brushSize:5});
toolGroup.setToolActive(strBrushCircle, { bindings: [ { mouseButton: cornerstone3DTools.Enums.MouseBindings.Primary, }, ], });
toolGroup.setToolEnabled(segmentationDisplayTool.toolName);

segmentation.addSegmentations([ {segmentationId, representation: { type: cornerstone3DTools.Enums.SegmentationRepresentations.Labelmap, data: { volumeId: segmentationId, }, }, }, ]);
const segmentationRepresentationUIDs = await segmentation.addSegmentationRepresentations(toolGroupId, [ {segmentationId, type: cornerstone3DTools.Enums.SegmentationRepresentations.Labelmap,}, ]);
segmentation.activeSegmentation.setActiveSegmentationRepresentation(toolGroupId,segmentationRepresentationUIDs[0]);

The current behavior

Unable to view contours across orthographic views

But able to view labelmap across orthographic views

The expected behavior

Behaviour of contours should be similar to that of labelmap

OS

MacOS v14.2 (Sonoma)

Node version

20.8.0

Browser

Brave (Version 1.66.118)

[sedghi]

This is expected behavior. Contours are based in 2D space, while labelmaps are 3D.

[prerakmody]

Thanks for the response.

Are there any plans in the roadmap to ensure that the 2D contours (which have an internal (x,y,z) representation ?) are also made visible in 3D space? Given that this is a feature available in commercial tools (e.g. MIM, ReLU, Eclipse, Raystation etc), having such a feature in open-source tools makes them more ecologically valid.

Maybe I can change this to a feature request?

[sedghi]

Theoretically, you cannot display x, y, z coordinates and a plane without assuming some thickness for the point and the plane. It's impossible to make the plane pass through that exact Z coordinate due to floating-point errors. All these tools make certain assumptions.

We already have labelmaps, which are volumetric, and you can use brush tools and other editing tools to modify the segmentation mask. The ReLU example was interesting; if you watch closely, they have the labelmap (edged mask) and then generate a contour around it. This allows you to edit the contour, and the changes apply back to the labelmap.

I'm not sure about MIM, but the smooth edges on their segmentation suggest that the internal representation is labelmap-based (volume and voxel-based). However, rendering is likely done via contours. The same process occurs when the contour is updated: the voxel map is edited, and then the contour is regenerated, I assume

Alternatively, you can create surfaces from contours, cut through the surface, and display it as contours again. This is similar to what we have with our polyseg converters, although it's not yet mature enough.

Another approach is to generate or edit a contour, then run it through a contour-to-surface conversion. On other viewports, we could cut through that surface. This is certainly possible, although it requires at least two slices with the contour since we can't generate the surface from a single contour plane. This approach is most attractive to me in the short term, as we already have the infrastructure for it.

[prerakmody]

Alternatively, you can create surfaces from contours, cut through the surface, and display it as contours again. This is similar to what we have with our polyseg converters, although it's not yet mature enough.

Ideally, I would like to take this approach. How do you recommend we "cut" through a surface? I mean how would we create the plane used to cut through the surface?

[sedghi]

I have all the building blocks already. They are just commented out since I wasn't able to finish the task and had to move on to something else.

https://github.com/cornerstonejs/cornerstone3D-beta/blob/main/packages/tools/src/stateManagement/segmentation/helpers/clipAndCacheSurfacesForViewport.ts

https://github.com/cornerstonejs/cornerstone3D-beta/blob/main/packages/tools/src/stateManagement/segmentation/polySeg/Contour/utils/updateContoursOnCameraModified.ts

we will look into this in near future though

[prerakmody]

Great! So do you have a public roadmap for cornerstone3D? I was unable to find it.

[sedghi]

no we don't have a roadmap for cs3d, but we have for ohif, see ohif.org

[IrvingLu]

I have the same problem, my temporary solution now is to use OpenCV to convert labelmap to contour display. I have tried using Surface to convert to contour display, but the performance is too slow. If there is a better way, please let me know. Thank you. And if I use contour display, I don't know how to modify my brush tool

[prerakmody]

Hi, Does the 2.x (https://www.cornerstonejs.org/docs/migration-guides/2x/tools/) release have an affect on this feature request.

I mean will we see edits to a contour reflect across different views?

[sedghi]

No, it actually facilitates it. I will get back to contour to surface cuts soon. But i pushed a lot of changes in this file packages/tools/src/tools/displayTools/Contour/contourDisplay.ts that should does the basic job of conversion, you can try it out

[prerakmody]

Hi, I used v2.2.15 and attempted cornerstone3DTools.segmentation.addSegmentationRepresentations() with cornerstone3DTools.Enums.SegmentationRepresentations.Contour

• However, I still only see the contour in the cornerstone3D.Enums.OrientationAxis.AXIAL view (see videos below where one has data with SegmentationRepresentations.LabelMap and the other with SegmentationRepresentations.Contour )
• I attempted to replicate polysegwasmvolumelabelmaptocontour from packages/tools/examples/PolySegWasmVolumeLabelmapToContour/index.ts

---

• SegmentationRepresentations.LabelMap https://github.com/user-attachments/assets/f8b35088-56f3-4fc3-9119-48672b317d69

• SegmentationRepresentations.Contour https://github.com/user-attachments/assets/a8322362-dd1a-49c7-a028-c6d4bcd0c4a5

To reproduce

I used the following code (look at last line of Step 3)

// --------------------- Step 1 - fetch the DICOM data ---------------------
const client = new dicomWebClient.api.DICOMwebClient({
    url: searchObj.wadoRsRoot
});

const arrayBuffer = await client.retrieveInstance({
    studyInstanceUID: searchObj.StudyInstanceUID,
    seriesInstanceUID: searchObj.SeriesInstanceUID,
    sopInstanceUID: searchObj.SOPInstanceUID
});
const dicomData = dcmjs.data.DicomMessage.readFile(arrayBuffer);
dataset         = dcmjs.data.DicomMetaDictionary.naturalizeDataset(dicomData.dict); 
dataset._meta   = dcmjs.data.DicomMetaDictionary.namifyDataset(dicomData.meta);

if (dataset.Modality === config.MODALITY_SEG){
    // --------------------- Step 2 - Load pixel array ---------------------
    const generateToolState = await cornerstoneAdapters.adaptersSEG.Cornerstone3D.Segmentation.generateToolState(
        imageIds,
        arrayBuffer,
        cornerstone3D.metaData
    );  

    // --------------------- Step 3 - Create c3D segmentation volume (and addSegmentations() and addSegmentationRepresentations()) ---------------------
    const derivedVolume = await cornerstone3D.volumeLoader.createAndCacheDerivedLabelmapVolume(config.volumeIdCT, {volumeId: segmentationId,});
    cornerstone3DTools.segmentation.addSegmentations([{ segmentationId:mySegmentationId, representation: { type: cornerstone3DTools.Enums.SegmentationRepresentations.Labelmap, data: { volumeId: mySegmentationId, }, }, },]);
    config.viewPortIdsAll.forEach((viewPortId) => {
        cornerstone3DTools.segmentation.addSegmentationRepresentations(viewPortId, [ { segmentationId:segmentationIdParam, type: cornerstone3DTools.Enums.SegmentationRepresentations.Contour, options: { polySeg: { enabled: true, }, }, }, ]);
    }); 

    // --------------------- Step 4 - Set the labelmap data ---------------------
    await derivedVolume.voxelManager.setCompleteScalarDataArray(new Uint8Array(generateToolState.labelmapBufferArray[0]));

    // --------------------- Step 5 - Add the segmentation to the 3D viewer ---------------------
    await cornerstone3DTools.segmentation.addSegmentationRepresentations(config.viewport3DId, [
        {
            segmentationId:mySegmentationId,
            type: cornerstone3DTools.Enums.SegmentationRepresentations.Surface,
            options: {polySeg: {enabled: true,},},
        },
    ]);
}

And this is how I defined the viewports

async function setRenderingEngineAndViewports(){

    const renderingEngine = new cornerstone3D.RenderingEngine(renderingEngineId);

    // Step 2.5.1 - Add image planes to rendering engine
    const viewportInputs = [
        {element: axialDiv     , viewportId: axialID     , type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC, defaultOptions: { orientation: cornerstone3D.Enums.OrientationAxis.AXIAL},},
        {element: sagittalDiv  , viewportId: sagittalID  , type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC, defaultOptions: { orientation: cornerstone3D.Enums.OrientationAxis.SAGITTAL},},
        {element: coronalDiv   , viewportId: coronalID   , type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC, defaultOptions: { orientation: cornerstone3D.Enums.OrientationAxis.CORONAL},},
        {element: axialDivPT   , viewportId: axialPTID   , type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC, defaultOptions: { orientation: cornerstone3D.Enums.OrientationAxis.AXIAL},},
        {element: sagittalDivPT, viewportId: sagittalPTID, type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC, defaultOptions: { orientation: cornerstone3D.Enums.OrientationAxis.SAGITTAL},},
        {element: coronalDivPT , viewportId: coronalPTID , type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC, defaultOptions: { orientation: cornerstone3D.Enums.OrientationAxis.CORONAL},},
        {element: viewport3DDiv, viewportId: viewport3DId, type: cornerstone3D.Enums.ViewportType.VOLUME_3D   , defaultOptions: { background: cornerstone3D.CONSTANTS.BACKGROUND_COLORS.slicer3D},},
    ]
    renderingEngine.setViewports(viewportInputs);
    
    // Step 2.5.2 - Add toolGroupIdContours to rendering engine
    const toolGroup = cornerstone3DTools.ToolGroupManager.getToolGroup(toolGroupIdContours);
    viewPortIdsAll.forEach((viewportId) =>
        toolGroup.addViewport(viewportId, renderingEngineId)
    );

    // Step 2.5.3 - Add toolGroupId3D to rendering engine
    const toolGroup3D = cornerstone3DTools.ToolGroupManager.getToolGroup(toolGroupId3D);
    toolGroup3D.addViewport(viewport3DId, renderingEngineId);

    // return {renderingEngine};
}

[sedghi]

I'll revisit this, but unfortunately, it's not a priority right now.

[prerakmody]

Thanks for the response. I see in the latest OHIF Roadmap that there is no mention of labelmap --> contours yet. So I guess folks looking for this feature will wait till at least till Q2 2025.

I'm curious if you will conduct user polls for Q2 2025 roadmap? But some very interesting features otherwise for Q1 2025 like LabelMap interpolation and SAM-based segmentation! Best of luck!

[sedghi]

Yes it will be after this iteration. So likely Q2, Q3 2025

[prerakmody]

Hi @sedghi, Any plans to do this in Q2 of this year?

[sedghi]

Yes very soon