Coordinate systems and new coordinate transformations proposal

[bogovicj]

This PR has four main contributions:

• The existing axis specification into coordinateSystems - named collections of axes.

• coordinateTransformations now have "input" and "output" coordinate systems.

• Adds many new useful type of coordinate transformations.

  • informative examples of each type are now given in the spec.

• Describes the array/pixel coordinate system (origin at pixel center)

  • as agreed in issue 89 (see below)

• Adds a longName field for axes

  • Also nice for NetCDF interop

See also:

• https://github.com/ome/ngff/issues/84
• https://github.com/ome/ngff/issues/94
• https://github.com/ome/ngff/issues/101
• https://github.com/ome/ngff/issues/89
• https://github.com/ome/ngff/issues/142

[github-actions[bot]]

Automated Review URLs

• render latest/index.bs
• diff latest modified

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-ngff-community-call-transforms-and-tables/71792/1

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/intermission-ome-ngff-0-4-1-bioformats2raw-0-5-0-et-al/72214/1

[bogovicj]

@sbesson

I really appreciate you're looking over this and for the helpful comments and feedback.

In this context, what is the goal of the transform-details.bs page included in the current PR i.e. would we consider splitting the document into multiple sections?

Initially, I had in mind to put lots of content in the transform-details.bs page. Mostly examples, discussions, and common use-cases. Semi-recently, I had a change in heart, and felt that some (most?) of it could go in the main document. I could still be convinced otherwise.

So to summarize, right now I have in mind to throw away transform-details, but could be convinced to migrate some of the content there (e.g. all the examples), if we decide its more appropriate there.

[ivirshup]

Couple questions about coordinate spaces for non-image elements.

Specifying a coordinate space for coordinate data

For something like point data or polygons, the coordinates of the points will be defined for some coordinate space. Any suggestions on how that would be annotated? Right now there is just a list of coordinate spaces that could apply, how do we say "these coordinates are for this specific coordinate space"?

Mapping coordinates into space with channels

If I have FISH-like data + histology, I could have:

• An image in a coordinate system of

{
    "name":"image-space",
    "axes": [ 
        {"name": "x", "type": "space"},
        {"name": "y", "type": "space"},
        {"name": "c", "type": "channel"} 
    ]
}

• Points in a coordinate system:

{
    "name":"point-space",
    "axes": [ 
        {"name": "x", "type": "space"},
        {"name": "y", "type": "space"},
    ]
}

How do you think we could indicate that "x", "y" axes are shared between these coordinate systems? How can we say that these points can be plotted on top of this image?

[LucaMarconato]

My proposal is to specify explicitly a space in which the various elements live, as opposed to say that the points live in the space of the image, or viceversa. This could be done by extending the concept of "array" space (valid for images and labels) to all the other spatial elements (i.e. points, and in the future polygons, etc.).

The array space acts as a default "local" space for images and labels, and this makes possible to specify a transformation to one (or more) shared spaces by listing the transformations from the local space to the shared. Having a default local space for points would make this possible also for those elements.

In practical terms, this could be a way to describe a datasets with two samples that don't share space information and so that we don't want to align in space.

{
   "coordinateSystems":[
      {
         "name":"sample0",
         "axes":[
            {
               "name":"x",
               "type":"space"
            },
            {
               "name":"y",
               "type":"space"
            },
            {
               "name":"c",
               "type":"channel"
            }
         ]
      },
      {
         "name":"sample1",
         "axes":[
            {
               "name":"x",
               "type":"space"
            },
            {
               "name":"y",
               "type":"space"
            },
            {
               "name":"c",
               "type":"channel"
            }
         ]
      }
   ],
   "coordinateTransformations":[
      {
         "name":"sample0_points",
         "type":"identity",
         "input":"my_storage/points0",
         "output":"sample0"
      },
      {
         "name":"sample1_points",
         "type":"scale",
         "scale":[
            1.0,
            1.2
         ],
         "input":"my_storage/points1",
         "output":"sample1"
      },
      {
         "name":"sample0_image",
         "type":"identity",
         "input":"my_storage/image0",
         "output":"sample0"
      },
      {
         "name":"sample1_image",
         "type":"translation",
         "translation":[
            10.0,
            10.0,
            1.0
         ],
         "input":"my_storage/image1",
         "output":"sample1"
      }
   ]
}

[bogovicj]

@ivirshup

If I have FISH-like data + histology

How do you think we could indicate that "x", "y" axes are shared between these coordinate systems?

This is a nice example, and is largely what I see identity being used for - to indicate that some coordinate system (or subset thereof is "the same as" another. But I've defined identity to be invertible (which I like), so to make it go 3d-2d it needs some help, and byDimension can do the job.

An alternative is to use an affine, which is not required to have equal input and output dimensions. In this case, we need a 4x2 matrix.

"coordinateSystems" : [
  {
      "name":"image-space",
      "axes": [ 
          {"name": "x", "type": "space"},
          {"name": "y", "type": "space"},
          {"name": "c", "type": "channel"} 
      ]
  }
  {
      "name":"point-space",
      "axes": [ 
          {"name": "x", "type": "space"},
          {"name": "y", "type": "space"},
      ]
  }
],
"coordinateTransformations" : [
  {
    "name" : "option 1 - using identity by dimension",
    "type": "byDimension",
    "input" : "image-space",
    "output" : "point-space",
    "transformations": [
       { "type" : "identity", "input" : ["x","y"], "output": ["x","y"] }
    ]
  },
  {
    "name" : "option 2- using affine",
    "type": "affine",
    "input" : "image-space",
    "output" : "point-space",
    "affine" : [ 1,0,0,0, 
                 0,1,0,0]
  }
]

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-ngff-community-call-transforms-and-tables/71792/11

[jni]

I just wanted to note down my main idea during the OME community meetings (thanks again to the organisers!), which is that coordinate systems should have types ("space" vs "time" vs "channel" vs "space-frequency"), but not units. The units should go on the transforms. Viewers and other tools consuming this metadata can then work in whatever unit is convenient to them, as long as that unit is compatible with the type of the space.

[ivirshup]

@jni, I'm not sure I totally understood your argument for putting units on the transforms instead of the spaces during the call, but it was also quite late for me 😅. Is this the main argument for it? And is there prior art for putting units on the transforms?

Viewers and other tools consuming this metadata can then work in whatever unit is convenient to them, as long as that unit is compatible with the type of the space.

To me, this seems like an easy solve from the viewer side. It can just define a new coordinate space and a scaling transform to get there. I would imagine you'd effectively also need this for your proposal?

---

Just to mention it here, I had brought up a point in favor of keeping the unit on the axes. While the base coordinate system on images are i.e. (e.g. pixel-space), we will have data types where this isn't the case. With point and polygon data we'll just have coordinates, and we'll want to know what {"x": 42, "y": 3.7} means in physical space. This is easy if we can say these points exist in:

{
  "axes": [
    {"name": "x", "units": "micrometers"}, 
    {"name": "y", "units": "micrometers"}
  ]
}

[LucaMarconato]

I also think that the units should not belong to the transformation because I see affine transformations as unitless. For instance if you scale by 2 it doesn't matter which units are involved, the transformation would act in the same way. Also I don't see convenient to include units to those transformations that are just injecting a lower dimensional space into a bigger one, like this transformations that maps a labels object (segmentation masks) into the cyx space of an image, that is, it goes from the pixel space (axes with array type) to a shared cyx space.

x' = x
y' = y
c' = 0

These type of functions act like an identity on a restricted codomain, which having a unit specified here would feel unnatural.

[LucaMarconato]

I have to correct myself, translations are an example of affine transformations that are not unitless, but many transformations are still defined without units (scale, rotations, transformations moving the axes, bydimension, etc.)

[bogovicj]

Thanks to everyone who attended the community call last week. A few critiques / questions arose in the discussions:

mapAxis vs mapIndex

@dzenanz suggested to remove mapIndex because its purpose overlaps entirely with mapAxis (direct mapping of input to output axes - permutations and projections)

• Someone else agreed but I forgot to write down their name
• I prefer mapAxis to mapIndex but chose to keep both in the proposal because some people at a hackathon preferred mapIndex

choosing between transformations

What to do of more than one transformation have the same input and output spaces (which to use?)

• Choose the "first" one, or allow selection by name (if names provided)
• A similar approach for choosing among options is already described for multiscales
• I will add text describing this possibility

representing paths

This also came up with respect to the tables proposal, so discussion has been forked into its own issue https://github.com/ome/ngff/issues/144

where do units belong?

@jni asked why units belong with axes + coordinateSystems and not say, with the tranformations. The idea being that viewers and tools should be free to work in whatever units they want and not be forced into something by the coordinate system. I see value in making dimensions / axes "identical up to changes of units" (~=) e.g. a spatial axis of unit "mm" ~= a spatial axis of unit "nm". I also completely agree that consuming software and users should not be forced into working with any particular units.

That is a reasonable choice to me, but here's why I like the current approach (with coordinate systems having units).

It semantically matches my idea of "coordinates"

For me, coordinates are the number of units away from the origin of a point. So having a coordinate without a unit doesn't make much sense. Meaning that, if coordinate systems don't have units, then one will generally (always?) have to specify units.

// if coordinate systems DO NOT have units
getImageValueAt( coordinateSystem, point, units )

// if coordinate systems DO have units
getImageValueAt( coordinateSystem, point )

It expect it would also mean that point annotations / rois etc would have to store their numeric coordinates, the coordinate system name, and their units. Whereas if coordinate systems have units attached, then any points in that coordinate system must be understood to have those units.

It encourages being explicit about transformations

Implementations will need to transform between units if they ever need to work with different units. If an implementation wants to work with "the same" coordinate system but with new units (and write it to a zarr store), the spec as written will force it to make a whole new coordinate system with a new name, and with the transformation between them. This is the price we pay for the "convenience" of not having to specify units above.

This relates to @ivirshup 's point above that consuming software are free to work in whatever units they want after scaling appropriately. To make extra clear: the spec is only strict about units on coordinate systems that are serialized and obviously has nothing to say about how things are implemented. So a viewer is free to treat spatial axes of any unit type as "the same" if they so choose.

Aside

We should consider inheriting more unit information from unidata, and could set up some conventions / defaults to make being explicit unnecessary sometimes.

[ivirshup]

mapIndex/ mapAxis vs permuteIndex/ permuteAxes

Wanted to touch on this from the call. Either is fine, but prefer the permute* variant. It's probably because I've used API's where this kind of transformation was permutedims. I also think map is pretty overloaded, and by name alone one could think mapAxes does something like apply a function over an axis. Permute is pretty specific to what is being done.

Tbh, I would also prefer permuteDimensions/ permuteAxes since it's a permutation of the axes not of an axis.

[bogovicj]

@ivirshup

While I'm okay with permute*, as @andy-sweet mentioned in the call, "permutations" are 1-1, whereas these operations can map one input axis to several outputs. Maybe this is okay though.

(Naming is hard)

[ivirshup]

(Naming is hard)

Yeah...

Probably why the wikipedia page for permutations has a section on "Other uses of the term permutation"

It says the term we want is ordered arrangement, and arrangeDims arrangeAxes isn't so bad either.

[dzenanz]

I also like permuteAxes better than mapAxis. It is not an issue if it can do a little bit more than simple permutation.

[LucaMarconato]

Some comments on sequence. sequence seems to be the only place where transformations can appear without input and output specified. I think this is useful, because if am creating for instance an affine transformation by combining translations and scale, then I am operating in the same coordinate system, so one can simply apply the transformations one after each other.

There are nevertheless some edge cases that should be addressed:

1. mapAxis alone without input and output coordinate systems specified is not enough to define how the transformation operates
2. In the case in which the user specifies the input and output for the transformations composing a sequence I think that they should match (output of a transformation equal to the input of the next) and otherwise the transformation should be invalid
3. Some comments related to point 2. 3.1. If the user combines a translation and affine, as long as the dimensions match, no problem. 3.2. If the user combines a translation and a mapAxis, as long as the mapAxis has input and output specified, in theory there is no problem, but the implementation gets more complex. In fact, if input and output are not specified for translation, then the implementation needs to deduce that the output coordinate system of translation is the same as the input of translation, which is the same as the input of sequence. This type of reasoning should be implemented recursively, for the case in which a sequence contain a sequence. 3.3. If the user combines a translation and a affine, both defined without specifying input and output (but valid because with matching dimensions), and then also combines a mapAxis defining for this both input and output, this still is not enough to define the transformation. In fact there is no way to deduce the output coordinate system of the affine transformation unless it is explicitly defined.

My thoughts on this, some possible approaches.

1. Leaving things as it is. As long as the implementer is aware of these cases and that the implementation needs to infer the output coordinate systems in some cases, there is no problem and simply the user would get an error if a transformation is defined ambiguously. Pro: maximum flexibility of the specs. Cons: harder implementation and easier to make mistakes as a user since it's easy to pass an affine matrix to a sequence with the wrong order of axes.
2. Requiring input and output for some transformations. Transformations that can operate with a combination of input/output where input != output, in particular affine, mapAxis, sequence, byDimension would be required to be specified together with the input and output coordinate systems when used inside a sequence. Transformations when it is assured that input = output, that is identity, translation, scale, rotation, can be specified inside a sequence without making input and output explicit. Pro: specs still very flexible and less ambiguous. Cons: less clean specs, implementation probably as difficult as for 1..
3. Requiring input and output for all the transformations.. Pro: specs simple to understood, simple implementation, less error possibility for the users. Cons: verbosity, one has to specify all the coordinate systems composing a sequence (but it is also true that one could reuse the same coordinate system in all the steps).

The logic for the cases 1 or 2 is about ~100 lines of code (I have implemented something in between 1 and 2, you can find it here in the static functions that start with inferring_cs...), but maybe we want to go directly to case 3 and avoid complications.

EDIT: the code I linked is a wip Python implementation of the transforms specs, at the moment still requiring some rounds of code review. After it is clean I will make some example notebooks and it could be considered to be detached from the spatialdata repo or reused in ome-zarr-py. I will follow up on this here: https://github.com/ome/ome-zarr-py/issues/229