feat(SR): SCOORD3D point annotations support for stack viewport

[igoroctaviano]

• Support for SCOORD3D point annotations (just rendering / no hydration)
• Support for SCT finding site (rendering labels to point and panel)
• Support to render untracked findings to the measurement panel in a new section so the user can interact (jump to measurement) even if it's not being tracked
• Support to render the same measurement to a different image id by frame of reference
• Related PR: https://github.com/cornerstonejs/cornerstone3D/pull/950
• Issue: https://github.com/OHIF/Viewers/issues/1215

Context

Former PRs:

• https://github.com/OHIF/Viewers/pull/3631

#planar

Changes & Results

Testing

Checklist

PR

• [] My Pull Request title is descriptive, accurate and follows the semantic-release format and guidelines.

Code

• [] My code has been well-documented (function documentation, inline comments, etc.)

Public Documentation Updates

• [] The documentation page has been updated as necessary for any public API additions or removals.

Tested Environment

• [] OS:
• [] Node version:
• [] Browser:

[netlify[bot]]

✅ Deploy Preview for ohif-platform-docs ready!

Name	Link
🔨 Latest commit	0a985d1ea44e3081f65416db782493bcf4193e2a
🔍 Latest deploy log	https://app.netlify.com/sites/ohif-platform-docs/deploys/66981a34fc52950008a70907
😎 Deploy Preview	https://deploy-preview-3857--ohif-platform-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[netlify[bot]]

❌ Deploy Preview for ohif-dev failed. Why did it fail? →

Name	Link
🔨 Latest commit	0a985d1ea44e3081f65416db782493bcf4193e2a
🔍 Latest deploy log	https://app.netlify.com/sites/ohif-dev/deploys/66981a341846f600075d8adb

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (a2a0090) 44.44% compared to head (4d83134) 44.44%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3857   +/-   ##
=======================================
  Coverage   44.44%   44.44%           
=======================================
  Files          80       80           
  Lines        1332     1332           
  Branches      327      327           
=======================================
  Hits          592      592           
  Misses        587      587           
  Partials      153      153           

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a2a0090...4d83134. Read the comment docs.

[igoroctaviano]

@sedghi can you take a look at these changes and let me know what you think? This allows the rendering of scoord3d points by FOR. There's another PR here that is related: https://github.com/cornerstonejs/cornerstone3D/pull/950 I tried to keep the changes to a minimum without impacting existent functioanlity.

[cypress[bot]]

Passing run #4004 ↗︎

0

43

0

0

0

Details:

CR Update
Project: Viewers	Commit: 6033d3eda8
Status: Passed	Duration: 05:03 💡
Started: May 28, 2024 4:46 PM	Ended: May 28, 2024 4:51 PM

Review all test suite changes for PR #3857 ↗︎

[mrceresa]

Hi guys, very interested on this, which is the current status?

Thanks and regards

[igoroctaviano]

Hi guys, very interested on this, which is the current status?

Thanks and regards

@mrceresa This is currently waiting for review. This implementation targeted stack viewports. @sedghi proposed adding the same support to volume viewports instead (natural choice for scoord3ds) but right now the stack version is what we will go with since volume would require a little more effort to implement.

[fedorov]

Convert the referenced series into a volume viewport

@sedghi I think it might be quite beneficial to support these annotations in the stack viewport. If you have a point that is not on the slice plane, you will have to interpolate to the point position. In the acquisitions that are used for prostate MRI you will be dealing with very anisotropic voxel sizes (0.5x0.5x4mm), so if you interpolate, you can get artifacts. And also I think it is very important for clinical workflow to see the point on the original slices.

I think there is significant value in being able to project the point to the closest slice and show it in the stack viewport, with a disclaimer to the user that the point was projected.

What do you think?

[sedghi]

@fedorov To see the point on the original slices we should use Stack Viewport, but the issue/complexity here is that that annotation needs to be viewed in other series in the same FOR and in the same Slice I guess (no?), how do you imagine handling those without some projection?

I don't think we interpolate for volume viewports the logic is: if the annotation is within the same slice thickness of any viewport, that viewport will show it

import { vec3 } from 'gl-matrix';
import { CONSTANTS, metaData } from '@cornerstonejs/core';
import type { Types } from '@cornerstonejs/core';
import { Annotations, Annotation } from '../../types';

const { EPSILON } = CONSTANTS;

const PARALLEL_THRESHOLD = 1 - EPSILON;

/**
 * given some `Annotations`, and the slice defined by the camera's normal
 * direction and the spacing in the normal, filter the `Annotations` which
 * is within the slice.
 *
 * @param annotations - Annotations
 * @param camera - The camera
 * @param spacingInNormalDirection - The spacing in the normal direction
 * @returns The filtered `Annotations`.
 */
export default function filterAnnotationsWithinSlice(
  annotations: Annotations,
  camera: Types.ICamera,
  spacingInNormalDirection: number
): Annotations {
  const { viewPlaneNormal } = camera;

  // The reason we use parallel normals instead of actual orientation is that
  // flipped action is done through camera API, so we can't rely on the
  // orientation (viewplaneNormal and viewUp) since even the same image and
  // same slice if flipped will have different orientation, but still rendering
  // the same slice. Instead, we choose to use the parallel normals to filter
  // the annotations and later we fine tune it with the annotation within slice
  // logic down below.
  const annotationsWithParallelNormals = annotations.filter(
    (td: Annotation) => {
      let annotationViewPlaneNormal = td.metadata.viewPlaneNormal;

      if (!annotationViewPlaneNormal) {
        // This code is run to set the annotation view plane normal
        // for historical data which was saved without the normal.
        const { referencedImageId } = td.metadata;
        const { imageOrientationPatient } = metaData.get(
          'imagePlaneModule',
          referencedImageId
        );
        const rowCosineVec = vec3.fromValues(
          imageOrientationPatient[0],
          imageOrientationPatient[1],
          imageOrientationPatient[2]
        );

        const colCosineVec = vec3.fromValues(
          imageOrientationPatient[3],
          imageOrientationPatient[4],
          imageOrientationPatient[5]
        );

        annotationViewPlaneNormal = vec3.create() as Types.Point3;

        vec3.cross(annotationViewPlaneNormal, rowCosineVec, colCosineVec);
        td.metadata.viewPlaneNormal = annotationViewPlaneNormal;
      }
      const isParallel =
        Math.abs(vec3.dot(viewPlaneNormal, annotationViewPlaneNormal)) >
        PARALLEL_THRESHOLD;

      return annotationViewPlaneNormal && isParallel;
    }
  );

  // No in plane annotations.
  if (!annotationsWithParallelNormals.length) {
    return [];
  }

  // Annotation should be within the slice, which means that it should be between
  // camera's focalPoint +/- spacingInNormalDirection.

  const halfSpacingInNormalDirection = spacingInNormalDirection / 2;
  const { focalPoint } = camera;

  const annotationsWithinSlice = [];

  for (const annotation of annotationsWithParallelNormals) {
    const data = annotation.data;
    const point = data.handles.points[0];

    if (!annotation.isVisible) {
      continue;
    }
    // A = point
    // B = focal point
    // P = normal

    // B-A dot P  => Distance in the view direction.
    // this should be less than half the slice distance.

    const dir = vec3.create();

    vec3.sub(dir, focalPoint, point);

    const dot = vec3.dot(dir, viewPlaneNormal);

    if (Math.abs(dot) < halfSpacingInNormalDirection) {
      annotationsWithinSlice.push(annotation);
    }
  }

  return annotationsWithinSlice;
}

[fedorov]

To see the point on the original slices we should use Stack Viewport, but the issue/complexity here is that that annotation needs to be viewed in other series in the same FOR and in the same Slice I guess (no?), how do you imagine handling those without some projection?

The idea is to project, and indicate to the user that the point is not the original one, but has been projected.

I don't think we interpolate for volume viewports the logic is: if the annotation is within the same slice thickness of any viewport, that viewport will show it

I do not understand the above. "If the annotation is within the same slice thickness of any viewport" - what does this mean?

[sedghi]

maybe we can meet and discuss?

[fedorov]

@igoroctaviano do we have an instance that can be pointed to a Google DICOM Store to test this functionality?

[igoroctaviano]

@fedorov We don't but you can quickly test it locally using your OIDC config and http://localhost:3000/projects/idc-sandbox-000/locations/us/datasets/idc-dicom-test-inventory/dicomStores/test-samples/study/1.3.6.1.4.1.14519.5.2.1.7310.5101.860473186348887719777907797922.

In my last discussion with the OHIF team, they mentioned that this PR would need some modifications to be merged, such as adding a new Cornerstone scoord3d point tool and adapter to cornertone instead of using OHIF's DICOM SR display tool. Unfortunately, we don't have the time/resources to implement these changes at this moment.

[fedorov]

you can quickly test it locally using your OIDC config

I am not quite sure what this means. But in any case, since there is no plan to make this available any time soon, and it will be reimplemented anyway, it is not worth it to spend time testing. It is unfortunate that after investing so much into developing this functionality from the IDC side, it is still in this state ...

[igoroctaviano]

you can quickly test it locally using your OIDC config

I am not quite sure what this means. But in any case, since there is no plan to make this available any time soon, and it will be reimplemented anyway, it is not worth it to spend time testing. It is unfortunate that after investing so much into developing this functionality from the IDC side, it is still in this state ...

I agree. The PR got approved at some point but things changed on the OHIF side. I'll leave the PR open since SCOORD3D support might be another project's common interest.

[wayfarer3130]

Could you test this with the changes for Viewports for Video and WSI? Much of the same jump to point capabilities were added as a core part of CS3D recently, and then supported in OHIF using the setViewReference capabilities. I'm not sure how much is still needed, and what other parts are still needed. The viewport PR is https://github.com/OHIF/Viewers/pull/4182 Could we arrange some time to meet and discuss this?