ANTsX
Feature Idea: Possible Memory Optimization for Multi-Stage Registrations
I noticed today that for cases where I do registrations like this: --transform Rigid.... --transform Similarity.... --transform Affine.... --transform Affine....
For each of the stages it appears that the same files are being loaded from disk into memory (the --metric Mattes pairs for each stage).
Considering these are the same files for most cases, I was wondering whether there's an optimization to be had for recognizing the same files are being used, and to re-use the already loaded ITK image objects.
This could reduce memory usage and IO time.
Alternatively, files should be read once needed, rather than all at the start, and after the stage of interest, should be freed so they don't use memory when being unused.
It's possible that it might help, but the code has all been written to expect a fresh copy of the image at each stage, so there's potential for caching to introduce some nasty bugs.
I'd need to see some compelling use cases with substantial performance benefits before considering this.
Sorry, I wasn't quite clear in the original description.
If I increase the number of affine stages, the memory load on the registration goes up! This is completely unexpected since after each stage those files aren't used again. Something, somewhere isn't freeing the moving/fixed/masks from the prior stages.
I agree caching is probably the wrong way to go.
I am trying to build a high-resolution template using aMCH2.sh. Individual volumes are big (774x723x1418) at the final stage of the registration, memory consumption goes almost 140GB, crashing in the nodes with 128GB of RAM. I am already using single precision (which is the default I believe). Is there anything I can do to bring the memory consumption below 128GB (beyond resampling)? This is the template call I used.
antsMultivariateTemplateConstruction2.sh -d 3 -a 0 -i 1 -c 1 -p "export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=48" -z /home/maga/DataSets/rat/full_resolution/ref/rattemplate0.nii.gz -q 1000x500x200x70 -n 0 -o rat_full_ /home/maga/DataSets/rat/full_resolution/*.nrrd
aMCH.sh probably uses less memory but switching probably won't get you below the desired memory threshold. You might try building a low-resolution version of the template first and then incorporate elements of template building by hand using the full resolution images.
Leaving a note for myself for when I have more development time. We can partially fix this in the registration scripts by switching to a two-call registration where we run the affine, save, and use as the initialization for the next command.