Clarify convention for `inverseOf` transformation in rfc5

[jo-mueller]

Breakout issue from this discussion over at #350 .

There's been a lot of confusion about how the directionality of transforms is defined in usecases involving non-invertible transformations wrapped inside an inverseOf transformation, i.e.

{
    "type": "inverseOf",
    "transformation" : {
        "type": "displacements",
        "path": "/path/to/displacements",
    },
    "input": "input_image",
    "output": "output_image",
}

For once, the problematic part is, that the proposal states that transformations [...] represent functions from points in the input space to points in the output space. However, this would clearly not be the case for a wrapped, inverted transformation. A solution would be to swap the input/output fields of the transformation, but from a registration, point of view, this obstructs how a registration transformation was originally called and computed.

In order to avoid blocking rfc5 going forward, we will continue the discussion here.

AI-generated synopsis

Core Issue

The discussion centers on the design and necessity of the inverseOf transformation type in the RFC5 specification, particularly its relationship to image registration tools like elastix. The main points of contention are:

• Directionality Confusion: RFC5 mandates that transformations are written in the forward direction (input → output). However, tools like elastix often return transformations in the opposite direction (fixed → moving image), which can be confusing.
• Purpose of inverseOf: The inverseOf transformation was introduced to allow users to store "backward" transformations (e.g., from elastix) as forward transformations, while signaling to implementations that they are actually inverses. This is intended to maintain interoperability with registration tools.
• Consistency vs. Flexibility: The current definition of inverseOf reverses the usual meaning of input and output fields compared to other transformations, which @dstansby argues is inconsistent and confusing.

---

Key Arguments

Perspective 1

• Consistency: The inverseOf transformation is the only one where input and output are defined differently (i.e., the transformation maps from output to input). This breaks the RFC5 convention that all transformations map from input to output.
• Simplicity: If a transformation is defined as Transform(..., input=image_A, output=image_B), elastix could interpret image_A as the fixed image and image_B as the moving image, without needing inverseOf. The current design adds unnecessary complexity.
• Proposed Fix: Align inverseOf with the rest of the RFC by making it represent a function from input to output, and use metadata or documentation to clarify that the contained transformation is actually the inverse.

Perspective 2

• Interoperability: The inverseOf field is a pragmatic solution for tools like elastix, which return transformations in the "backward" direction. Without inverseOf, users would need to manually invert transformations, which is error-prone and not always possible (e.g., for non-invertible transforms).
• Clarification: The current design allows users to express, "This transformation is written as forward, but it’s actually backward." This is critical for correct interpretation by registration libraries.
• Compromise: Suggests rewording the spec to clarify that inverseOf is for cases where the transformation was computed as a backward mapping, but is stored as a forward transform for consistency with RFC5.

---

Proposed Resolutions

1. Align Definitions: Redefine inverseOf so that it maps from input to output (like all other transformations), but add a note that the contained transformation is the inverse. This would make the API consistent.
2. Documentation: Improve the spec text to clearly explain the use case for inverseOf and provide examples for tools like elastix.
3. Further Discussion: Move detailed resolution to a dedicated issue or synchronous discussion to avoid blocking the PR.

---

Open Question: Should the spec prioritize consistency in field definitions (input/output) or flexibility for specific use cases (like elastix)? The group leans toward finding a middle ground that maintains clarity and interoperability.

[thewtex]

It is important to maintain clarity regarding transformation directionality and interpretation:

1. In elastix, transformations always map locations from the input space to the output space.
2. Transformations (in general) may be applied in different contexts such as image resampling, point set transformation, mesh deformation, or rasterization for visualization. Each of these applications may require a different mapping direction to reach the same physical coordinate system.
3. The terms “forward” and “inverse” (or “forward” and “backward”) are only meaningful relative to one another. Their interpretation depends entirely on the context of use.

Directionality Confusion: RFC5 specifies that transformations are expressed in the forward direction (input → output). However, tools like elastix often return transformations in the opposite direction (fixed → moving image), which can be confusing.

This is a common source of misunderstanding. Transformations in elastix are consistently defined from input space to output space. When resampling a moving image on the grid defined by a fixed image, the transformation evaluated during sampling maps from fixed space to moving space. This does not indicate that the transformation itself is defined in the opposite direction—it simply reflects the way the transform is applied during resampling.

Interoperability: The inverseOf field is a pragmatic solution for tools like elastix, which return transformations in the "backward" direction. Without inverseOf, users would need to manually invert transformations, which is error-prone and not always possible (e.g., for non-invertible transforms).

The need to represent inverse relationships is not specific to elastix or image registration. It applies more broadly wherever transformations are composed, applied, or reused in different contexts.

The current “bijection” transform model adequately represents a forward/inverse pair. A “method” metadata attribute, similar to the multiscales method, could be added for provenance if necessary. Removing the “inverseOf” reference would simplify the model and help avoid ambiguity regarding directionality.

[jo-mueller]

Hi @thewtex , thanks for chiming in, it's helpful to have the opinions from the field.

This is a common source of misunderstanding. Transformations in elastix are consistently defined from input space to output space.

It is confusing. I just realized during writing the RFC that elastix (which only serves as a placeholder example here) "claims" to express transformations as inverse, but actually returns the forward transform (only checked for invertible transforms). Do you know how tools like elastix (or others) would handle this for a non-invertible transform?

The current “bijection” transform model adequately represents a forward/inverse pair. A “method” metadata attribute, similar to the multiscales method, could be added for provenance if necessary. Removing the “inverseOf” reference would simplify the model and help avoid ambiguity regarding directionality.

Are you proposing to allow a bijection written with only one of forward or inverse? I think I could get behind that - would be a comparatively simple change 👍

[thewtex]

elastix (which only serves as a placeholder example here) "claims" to express transformations as inverse, but actually returns the forward transform (only checked for invertible transforms). Do you know how tools like elastix (or others) would handle this for a non-invertible transform?

@jo-mueller what do you mean?

Again, "forward" and "inverse" only have meaning relative to each other and in a provided context.

Also, the direction is not related to whether a transformation is invertible.

And, while it is easier to find the inverse of translation and affine transforms through the negative and matrix, it is usually possible, with more computation, to also find the inverse of displacement fields because they are usually diffeomorphic.

Are you proposing to allow a bijection written with only one of forward or inverse?

No :-). bijection needs to have both forward and inverse. When using the transformation, it is useful to have the pair. And, for example, the re-computation a displacement field inverse every time it is needed is not practical.

I am proposing to remove inverseOf.