[Early WIP] - Lazy resampling

[atbenmurray]

Lazy Resampling - deep refactor (see #4855)

The goal is to provide less memory-intensive, faster and higher quality spatial resampling by adopting the best practice of traditional graphics pipelines.

This change impacts on most of the spatial transform code, and so is also an opportunity to refactor said code to detangle what have become quite complex spatial transforms. While #4911 demonstrates that #4855 is possible, this PR is trying to find a way to decouple elements of this solution from one another and dramatically simplify the task of implementing new transforms in future and by third parties.

Transform refactoring

Due to the complexity of existing transforms and the need to avoid adding more complexity, this refactor looks at the transforms from first principles, attempting to decouple the definition of the transform from its application. This can be seen in the three layers of transform implementation, functional, array and dictionary:

Functional

Functional transforms now have a single responsibility, to generate the definition of the transform.

Array

Array transforms now wrap the Functional transforms, and then decide whether to push the transform to a pending queue or apply it immediately. They do this by descending from LazyTransform, which has a flag lazy_evaluation. This could be taken further; the pattern of whether to push to pending or apply immediately could be part of LazyTransform functionality.

Dictionary

Dictionary transforms wrap Array transforms, applying them once for each key. There is potential here to further reduce the code by having a standard 'apply to each specified key' function that can be called by subclasses.

The amount of code is dramatically reduced as a result, at least in the transforms reimplemented so far.

Compose compilers

As part of looking at this refactor, it is becoming obvious that there are a few places where the list of transforms to be executed are manipulated as they are being executed. Each place in the code that does this does so with the assumption that no-one else is doing it. As such, one compose manipulator will "win" and the others do not appear to be applied at all.

One way around this is to specify "compose compilers". These are functions that will take an input list of transforms and compile it to a new list of transforms. The benefit of compose compilers is twofold:

1. They reduce the number of places where handling of the compose is customised
2. They can be chained in order to meet the demands of multiple transform list customisations (lazy resampling and cached datasets, for example)

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

TODO:

Spatial transform implementation

See Table: Implementation by Spatial Transform for detailed progress:

• Implement functional versions of all spatial transforms
• Implement array versions of all spatial transforms
• Implement dictionary versions of all spatial transforms

LazyTransform and RandomTransform interfaces for 3rd party use

Interfaces that indicate capability without needing to adopt implementations:

• Implement LazyTransform interface
• Optionally reimplement RandomTransform interface for 3rd party use

Compose compilation

This is done if we go with compose compilers as a unified mechanism for lazy resampling, smart datasets, etc.:

• Compose compiler implementation
• Compose accepts compose compilers
• Compose compiler transforms
  • Cached / Persistent dataset transforms
  • Multi-sample transforms

MetaTensor updates

Optionally make MetaTensors able to share read only access to underlying Tensor for multi-sampling transforms:

• MetaTensor sharing underlying tensor

Lazy resampling compose

This is done if we go with augmenting compose and leaving smart datasets alone:

• Compose performs lazy resampling

Table: Implementation by spatial transform

#                                                                                               R
#                                                                                               a
#                                                                                               n
#                          S  R                                                                 d
#                          p  e                                         R  R                 G  G
#                          a  s                                         a  a           R  R  r  r        R
#                          t  a                       R        R        n  n           a  a  i  i        a
#                          i  m     O                 a        a        d  d           n  n  d  d        n
#                          a  p     r                 n  R     n     A  A  D        R  d  d  D  D        d
#                          l  l     i                 d  a     d     f  f  e        a  2  3  o  i  G  G  G       T
#                          R  e     e              R  R  n  R  A  R  f  f  f  R     n  D  D  s  s  r  r  r    I  r
#                          e  T  S  n              o  o  d  a  x  a  i  i  o  e     d  E  E  t  t  i  i  i    d  a
#                          s  o  p  t     R  R     t  t  R  n  i  n  n  n  r  s  A  A  l  l  o  o  d  d  d    e  n
#                          a  M  a  a     e  o     a  a  o  d  s  d  e  e  m  a  f  f  a  a  r  r  S  P  S    n  s
#                          m  a  c  t  F  s  t  Z  t  t  t  F  F  Z  G  G  G  m  f  f  s  s  t  t  p  a  p    t  l
#                          p  t  i  i  l  i  a  o  e  e  a  l  l  o  r  r  r  p  i  i  t  t  i  i  l  t  l    i  a
#                          l  c  n  o  i  z  t  o  9  9  t  i  i  o  i  i  i  l  n  n  i  i  o  o  i  c  i    t  t
#                          e  h  g  n  p  e  e  m  0  0  e  p  p  m  d  d  d  e  e  e  c  c  n  n  t  h  t    y  e

Functional                       X  .  X  X  X  X  X  n  n  n  n  n  n  n  n  n  .  n  n  n  .  n  .  .  n    X  X
Array forward                    X  .  X  X  X  X  X  X  X  X  X  .  .  .  .  n  .  .  .  .  .  .  .  .  .    X  X
Dictionary forward               X  .  .  X  X  X  .  .  .  .  .  .  .  .  .  n  .  .  .  .  .  .  .  .  .    .  X
Test forward                     f  .  .  f  f  f  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .  .
Array inverse                    .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .  .
Dictionary inverse               .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .  .
Test inverse                     .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .  .

Table: Implementation by Crop / Pad transform

#                                                  R           R
#                                                  a        R  a
#                                                  n        a  n
#                                                  d        n  d  R
#                                                  S        d  C  e
#                                      C           p        C  r  s
#                                      e           a     R  r  o  i 
#                                      n  C  R     t     a  o  p  z
#                                      t  e  a     i  C  n  p  B  e
#                                      e  n  n  R  a  r  d  B  y  W
#                             D        r  t  d  a  l  o  W  y  L  i  B
#                             i     S  S  e  S  n  C  p  e  P  a  t  o
#                       S     v     p  p  r  p  d  r  F  i  o  b  h  u
#                       p  B  i     a  a  S  a  S  o  o  g  s  e  P  n
#                       a  o  s     t  t  c  t  c  p  r  h  N  l  a  d
#                       t  r  i     i  i  a  i  a  S  e  t  e  C  d  i    C
#                       i  d  b     a  a  l  a  l  a  g  e  g  l  O  n    r
#                       a  e  l     l  l  e  l  e  m  r  d  L  a  r  g    o
#                       l  r  e  C  C  C  C  C  C  p  o  C  a  s  C  R    p
#                    P  P  P  P  r  r  r  r  r  r  l  u  r  b  s  r  e    P
#                    a  a  a  a  o  o  o  o  o  o  e  n  o  e  e  o  c    a
#                    d  d  d  d  p  p  p  p  p  p  s  d  p  l  s  p  t    d

Functional           .  .  .  .  .  .  .  .  n  n  n  .  n  n  n  .  .    X
Array forward        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    X
Dictionary forward   .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    X
Test forward         .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .
Array inverse        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .
Dictionary inverse   .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .
Test inverse         .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .    .

[wyli]

when we have a chain of lazy transforms, img.shape will become outdated, correct? because we don't actually run img_t = apply(img_t).

so I think accessing img.shape at this point is inaccurate, this should be something like metadata['spatial_shape'] where metadata should come from a previous lazy transform or img.shape if the previous transform evaluates eagerly.

and in this case, the input of this function should include a metadata or input_shape?

[wyli]

and it seems img is not really needed in this function (and in the other functionals as well), just need the up-to-date spatial shape information of it.

[atbenmurray]

Yep, I'm aware of the issue (chain of lazy transforms). I'm still trying to figure out the best way to handle when / how transforms to shape / extents occur. I'll hopefully have it sorted by the end of the day; just experimenting at the moment.

[atbenmurray]

I think that we should be able describe the extent change by concatenating the individual extent change operations at the point we perform the stacking of transforms; details to follow

[atbenmurray]

I've gone the way @wyli went in PR #4911. I thought I could do this by applying the accumulated transform to the extents as they are defined at that point in the chain, but there are a couple of scenarios that seem to break that.