ngff icon indicating copy to clipboard operation
ngff copied to clipboard

Metadata for color and contrast limits

Open tischi opened this issue 4 years ago • 19 comments

@joshmoore @will-moore

It would be great to add metadata specifications for colour and contrastLimits (aka displayRange).

Below an example of how I am opening the data with BigDataViewer and how I am setting the colours, thus showing the kind of information I would need:

OMEZarrS3Reader reader = new OMEZarrS3Reader( "https://s3.embassy.ebi.ac.uk", "us-west-2", "idr" );
SpimData image = reader.read( "zarr/v0.1/6001237.zarr" );
List< BdvStackSource< ? > > sources = BdvFunctions.show( image );
sources.get( 0 ).setColor( new ARGBType( ARGBType.rgba( 0,0,255,255 ) ) );
sources.get( 0 ).setDisplayRange( 0, 3000 );
sources.get( 1 ).setColor( new ARGBType( ARGBType.rgba( 0,255,0,255 ) ) );
sources.get( 1 ).setDisplayRange( 0, 3000 );
sources.get( 2 ).setColor( new ARGBType( ARGBType.rgba( 255,0,0,255 ) ) );
sources.get( 2 ).setDisplayRange( 0, 3000 );
sources.get( 3 ).setColor( new ARGBType( ARGBType.rgba( 255,255,255,255 ) ) );
sources.get( 3 ).setDisplayRange( 0, 3000 );

tischi avatar Dec 01 '20 08:12 tischi

Agreed! Currently we are exporting the OMERO metadata in all of the IDR sample images. You can see what the metadata looks like in the spec: https://ngff.openmicroscopy.org/latest/#omero-md and sample code is available in ome-zarr-py for setting contrast limits, etc: https://github.com/ome/ome-zarr-py/blob/master/ome_zarr/reader.py#L293

However, we don't yet consider the "omero" namespace as official. Let's discuss on this issue how best to migrate from "omero" to proper rendering information in the spec. :+1:

joshmoore avatar Dec 01 '20 08:12 joshmoore

Hi @d-v-b, Are you having display settings within the N5 openOrganelle images?

tischi avatar Feb 24 '21 13:02 tischi

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

imagesc-bot avatar Feb 24 '21 15:02 imagesc-bot

@tischi I haven't come to a decision for display settings that I feel super comfortable with yet. At the moment I don't put the display settings in n5 metadata -- instead, I have a large json document at the root directory of each dataset that contains consolidated metadata for all the datasets we want to display, and this is where the display settings live. But if there's a need for it I have no problem with putting the display settings in the dataset or group metadata.

My display settings are really simple: e.g. "displaySettings": {"contrastLimits": {"min": 0, "max": 0.006958113984893569}, "color": "white", "invertColormap": false}, and only openorganelle parses this datastructure. I would love to abandon this and use something more standard though.

d-v-b avatar Feb 24 '21 16:02 d-v-b

@d-v-b: is there one per channel? Otherwise, looks like there's a good deal of overlap. From https://ngff.openmicroscopy.org/latest/#omero-md:

"channels": [                         # Array matching the c dimension size
    {
        "active": true,
        "coefficient": 1,
        "color": "0000FF",
        "family": "linear",
        "inverted": false,
        "label": "LaminB1",
        "window": {
            "end": 1500,
            "max": 65535,
            "min": 0,
            "start": 0
        }
    }
],
"rdefs": {
    "defaultT": 0,                    # First timepoint to show the user
    "defaultZ": 118,                  # First Z section to show the user
    "model": "color"                  # "color" or "greyscale"
}

joshmoore avatar Feb 24 '21 16:02 joshmoore

As you can see in the links above, in OMERO we currently use start/end for the current rendering settings, as well as min/max for the range of rendering sliders for the client to display. The min/max is usually the intensity range of the pixels in the image. Maybe this shouldn't live in 'contrast limits' but in other metadata, but it can certainly be useful, e.g:

"window": {
            "end": 1500,
            "max": 65535,
            "min": 0,
            "start": 0
        }

will-moore avatar Feb 24 '21 16:02 will-moore

@joshmoore yes there's a lot of overlap :) Switching to the omero format is on my to-do list actually, I just haven't gotten around to it yet. It would also be interesting if neuroglancer could natively read this metadata, which would allow me to delete some code from openorganelle :)

d-v-b avatar Feb 24 '21 16:02 d-v-b

Also in OMERO, we support LUTs e.g. "lut": "cool.lut", but that relies on the server knowing what LUT that refers to and is probably outside the scope of this issue.

will-moore avatar Feb 24 '21 16:02 will-moore

Switching to the omero format is on my to-do list actually

:+1: but is it worth simplifying the above before you do that?

It would also be interesting if neuroglancer could natively read this metadata

Do we need to ping anyone else for that? Does neuroglancer already have a format it uses?

that relies on the server knowing what LUT that refers to

Yeah, that's likely another spec with an enum and then extensible LUT storage in an array as well.

joshmoore avatar Feb 24 '21 16:02 joshmoore

To my knowledge Neuroglancer does not use dataset metadata directly, but the developer (@jbms, who probably needs to be added to this conversation to weigh in) is pretty responsive and I think he would consider this kind of feature in-scope.

Overall I think the current display settings are fine, maybe with the exception of the active field -- what does this convey?

d-v-b avatar Feb 24 '21 16:02 d-v-b

Neuroglancer supports brightness contrast adjustment via the "invlerp" UI control, which is documented here: https://github.com/google/neuroglancer/blob/master/src/neuroglancer/sliceview/image_layer_rendering.md#invlerp-controls

Currently, as @d-v-b noted, Neuroglancer doesn't have a mechanism for these settings to be determined by the data source, but I think it could in principle be supported. Currently the default settings are just based on the full range of the data type, but they could instead be determined by the datasource with some modifications...

Another feature that could be added to Neuroglancer is a command for automatically setting the range based on the distribution of currently visible values, which might make it mostly unnecessary to even have a default.

jbms avatar Feb 24 '21 17:02 jbms

The active field is whether a channel is on/off. Since some images have many channels (e.g. over 50), you only want to turn on a small number at once.

will-moore avatar Feb 24 '21 17:02 will-moore

At the moment I don't put the display settings in n5 metadata -- instead, I have a large json document at the root directory of each dataset that contains consolidated metadata for all the datasets we want to display, and this is where the display settings live.

@d-v-b This is very interesting, because I was actually wondering whether that's in some sense the better approach. The idea would be that the display settings could in fact be part of specifications how to visualize collections of images. In other words: there is a the "raw data" and there are certain "views" (could be several) specifying how all the raw data may be nicely visualized, including layout specification (like HTM multi well plate) and display settings.

tischi avatar Feb 24 '21 17:02 tischi

@will-moore in that case active is a bit of a semantic outlier -- all the other display metadata describes how intensities of the image should be displayed, however active is describing something about the UI state of that application, which seems like it should be in a separate/ higher category of metadata that describes a "scene" or a "view".

@tischi consolidating metadata has a very practical advantage: I want to keep the number of HTTP requests low, so putting all the dataset metadata in one place means an application only has to make 1 GET request to get a full index of the datasets. Additionally, I wasn't fully comfortable at a conceptual level with putting display metadata on n5 arrays directly -- display metadata somehow feels very application-specific and not at all intrinsic to the data itself (unlike spatial transform information). I think your idea of multiple "views" of data with different display settings accords with this feeling. But this is just a feeling. :)

d-v-b avatar Feb 24 '21 17:02 d-v-b

@d-v-b I think those are quite valid points. Maybe we could work on a spec proposal along those lines to try to see whether we can convert our feelings into something substantial?

tischi avatar Feb 24 '21 18:02 tischi

@d-v-b The way I think of active is certainly a UI state, but this UI state describes how the image should be displayed. This is also true of intensity settings which can be controlled via the UI (e.g. via sliders). Both can be saved in a similar way and used to set the initial UI state (or rendering settings) in a different viewer. E.g. http://idr.openmicroscopy.org/webclient/img_detail/9836831/ shows the same 3 channels initially active as in https://hms-dbmi.github.io/vizarr/v0.1?source=https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836831.zarr

will-moore avatar Feb 24 '21 22:02 will-moore

@will-moore good point, that makes sense @tischi I'd be up for making a spec proposal... how do we do that?

d-v-b avatar Mar 01 '21 18:03 d-v-b

@d-v-b I feel this maybe became quite related to this issue: https://github.com/ome/ngff/issues/31 Because maybe we only want to specify display settings on the level of a "specific view" on the dataset? Also related, does it makes sense conceptually to think of MutableImageProperties and ImmutableImageProperties? Where the DisplaySettings would be part of the MutableImageProperties? At least in my Java code I currently have such classes.

(Personally, I have the issue right now that I have only time again from 22nd of March to really work on this; if that's not too late maybe could have a zoom in that week with a group of interested people)

tischi avatar Mar 02 '21 10:03 tischi

https://github.com/ome/ngff/issues/23#issuecomment-788163198 I'd be up for making a spec proposal... how do we do that?

The steps from my POV follow, but you can skip any of the former ones based on how much feedback you want before investing more time:

  1. Write down one or more proposed specifications (usually as sample JSON blocks initially) and give the pros and cons. That could be done here or image.sc. Usually I update the description of the issue as I make changes and keep a changelog, but that may be overkill.
  2. Open a PR against this repository which includes a more formal spec, including MAY/MUST/SHOULD differentiations, more examples, etc.
  3. Open a PR against one or more of the implementations (bioformats2raw, ome-zarr-py, zarr.js, bioformats, etc.) to show how the specification would work.
  4. Produce sample data that the community can examine. (We're happy to host this if it's of use, e.g. https://s3.embassy.ebi.ac.uk/idr/share/2021-Q1-3D/)

joshmoore avatar Mar 04 '21 15:03 joshmoore