Consolidated Resampling

[atbenmurray]

Introduction

Different types of spatial transform carry out different types of resampling on the dataset. A given transform can have any or all of the following modifications:

• In axis interpolation
• Off axis interpolation
• Sampling at pixel centers
• Sampling off pixel centers
• Affine transform-based transform description
• Grid-based transform description
• Various interpolation strategies, from nearest neighbor to gaussian sampling

There is a lot of code across the transforms that sets up and carries out the transform using one of the many layers and functions used as a transform backend. These are listed in the table below.

Timeline

Although creating generic Resample functionality could be done on 1.0 timescales, it is not likely that refactoring of spatial transforms could be done on this timescale. Furthermore, in the light of #4855, it is not obvious that we would want to refactor spatial transforms independent of making them lazily applicable. As such, this could be considered a part of that overall work.

Rationale for simplification

A general resample function could take a description of a transformation along with the additional flags needed to fully describe it. It could examine the transform in question and make decisions about what backend to use for resampling. The selection criteria are two-fold:

1. Select the resampling mechanism that is most accurate
2. Select the resampling mechanism that is cheapest

This information can be determined from a homogeneous affine transformation matrix or from a grid representation. This can be used to determine:

1. whether the transform is orthogonal to the axes or not
2. whether there is a scaling component

Along with the other parameters, such as mode (resampling mode) and padding_mode, the correct backend mechanism can be selected ensuring fast and effective resampling in the general case, and reducing the complexity of all spatial transforms, that are free to phrase their transforms as either matrices or grids as appropriate.

See also

• #4974 as a unified resample operation allows for more flexible operations in transforms
• #4855 as both this and and #4974 allow lazy resampling to be implemented in a simpler way than wiring it into the current transforms

Table of backends for spatial transforms

                           S O F R R Z R R R R R R   R A R   A R R   R R G R
                           p r l e o o o a a a a a   e f a   f a a   a a r a
                           a i i s t o t n n n n n   s f n   f n n   n n i n
                           c e p i a m a d d d d d   a i d   i d d   d d d d
                           i n   z t   t R R F A Z   m n A   n A D   2 3 D G
                           n t   e e   e o o l x o   p e f   e f e   D D i r
                           g a         9 t t i i o   l   f   G f f   E E s i
                             t         0 a a p s m   e   i   r i o   l l t d
                             i           t t   F         n   i n r   a a o D
                             o           e e   l         e   d e m   s s r i
                             n           9     i               G G   t t t s
                                         0     p               r r   i i i t
                                                               i i   c c o o
                                                               d d       n r
                                                                           t
                                                                           i
                                                                           o    
                                                                           n

don't transform images                                       x x x
----------------------------------------------------------------------------
monai.SpatialResample      x
tensor.permute               x
monai.Flip                     x
torch.interpolate                x
monai.AffineTransform              x
monai.ResizeWithPadOrCrop            x
torch.rot90                            x
monai.Rotate90                           x
monai.Rotate                               x
monai.Flip                                   x x
monai.Zoom                                       x
torch.grid_sample                                    x
m.AffineGrid->m.Resample                               x
m.RandAffineGrid->m.Resample                             x
Not applicable                                               . . .
m.RandDeformGrid->m.RandAffineGrid->t.interpolate                    x x
monai.Resample                                                           x
monai.GridDistortion                                                       x