Adding Images.jl support

[Tokazama]

I'm trying to update this package and add Images.jl support. I've gotten to the point where I can get include("./src/NIfTI.jl") working on Julia 1.0. However, I'm not entirely sure how to proceed with incorporating Image.jl support.

It's suggested here that image types have only two fields (properties and data). I figured I could proceed with one of the following strategies

1. The current header and extensions field could be combined into the properties field so that NIVolume <: AbstractImage.
2. Have some function that maps the current structure to a proper image type?
3. Different load options for including all of the header information (current) and another option that reads to and AbstractImage subtype.

Thanks in advance

[tknopp]

Ping @timholy on this subject.

My point of view is the following: I would like it if we could read a nifti file and the result will be an ImageMetadata object. several thing like image dimension and so on will be encoded into the AxisArray that the ImageMetadata wraps.

One issue I am unclear about is how the metadata should be presented. Will the resulting image object look similar if we load a Nifti or a DICOM file? Do we perform some "normalization" on the parameters?

The end goal from my perspective should be

im = load("file.ni")
save("file.dcm", im)

There can still be something like NIVolume <: AbstractImage which would, however, be only necessary for partial reading of files. The ImageMetadata would then be kind of a materialization.

[timholy]

@Tokazama, oh dear, sorry, those docs are ancient. I didn't even realize they were still part of the repository. The Images.jl homepage does link to https://juliaimages.github.io/latest/ which is current, but we should delete those docs.

As @tknopp says, using an ImageMetadata (which is basically the same thing as what you were referencing) could fix the issue. Normalization will be important if you want multiple file formats to recognize the format. The only format that returns consistent ImageMetadata now is NRRD, so perhaps check that for hints.

[Tokazama]

I've been going back and forth in my head a lot on this lately and I think the thing to do would be something like a NiftiImage

mutable struct NiftiImage
    data::AxisArray  # contains spacing, timing, orientation
   properties::Dict{String,Any}   # contains Nifti1Extensions, descrip field, intent_* fields, etc

I don't think we should even worry about tracking those fields that are unused in the NIfTI1 standard (eg, glmax, glmin, etc) since when writing the image those will probably just be given default values. So some save function would just check the properties for appropriate fields and extract sform stuff from the AxisArray.

I imagine implementing something similar for DICOMs would result in essentially identical structure for the data field. The properties field would be pretty variable depending on what people continue to add to the DICOM standard.

[timholy]

Assuming there's lots of Nifti/DICOM-specific stuff in the header, I think that's a really good plan. If one wants to then write it out as a DICOM one could use @requires and dispatch on the type.

[tknopp]

Do we really require a dedicated type? Of course dicom and nifti will have additional metadata but having everything resulting in an ImageMetadata object will make the handling much easier from the user perspective. We should introduce required parameters that are normalized when loading and denormalized when saving. This concept for nifti, dicom, nrrd, vtk, and further formats would be awesome.

[timholy]

Hmm, you're right, perhaps run-time checking is a better idea: there's nothing performance-sensitive about testing whether a given image is from a particular file type (you're not checking that on a pixel-by-pixel basis). What about the following standard for the ImageMetadata properties:

Dict{String,Any} with 2 entries:
  "fileformat" => "Nifti"
  "fileheader" => Dict{String,Any}("niftifield1"=>'a',"niftifield2"=>2)

Of course you could extract additional info into other named properties, and perhaps some of these could become semi-standard.

For biomedical image formats, I highly recommend leveraging AxisArrays. If you've not seen it, try

julia> using TestImages

julia> img = testimage("mri");

julia> print(summary(img))
3-dimensional AxisArray{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},3,...} with axes:
    :P, 0:1:225
    :R, 0:1:185
    :S, 0:5:130
And data, a 226×186×27 Array{Gray{N0f8},3} with eltype ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}}

This image is represented in PRS coordinates, hence the naming of the axes. And note the voxel separations encoded in the ranges.

[tknopp]

ping @hofmannmartin who is working together with me on this concept for a different format (MDF). I may also do this for the ISMRMRD format.

[Tokazama]

I don’t think it would really require a dedicated type.

I do think there should be a more modular framework that allows reading just header data. It's not uncommon to want some info from the header before importing the entire image array.

It would also make it easier for other packages to build on NIfTI.jl. I imagine that some would like GIfTI, CIfTI, etc support. This is essentially NIfTI with some other stuff in the extension.

[timholy]

NRRD supports version, header, keyvals, comments = parse_header(io). This pattern works if you know you're parsing a *.nrrd file. However, figuring out the file type is generically supported by FileIO's query, so it really just comes down to deciding what should happen subsequently. If we want something generic then we have to think about what kind of common information could be extracted.

[tknopp]

Maybe an "ImageFormats.jl" package is in order. Either this could bundle everything together ( inclusive testing conversions) or it could be the basis.

[tknopp]

What about the following standard for the ImageMetadata properties:

Dict{String,Any} with 2 entries: "fileformat" => "Nifti" "fileheader" => Dict{String,Any}("niftifield1"=>'a',"niftifield2"=>2)

I like this. We have three categories:

• First the things that can be encoded into the AxisArray.
• Then those parameter were we have some common name. For instance I am storing a rotation matrix in the image. We should have a normalized form here. These parameters could be "toplevel" in the dict. I think we can also agree on things like "subject", "study", and things like this.
• Finally, all the format specific things that go into the "fileheader" Dict.

[timholy]

For instance I am storing a rotation matrix in the image. We should have a normalized form here.

It already exists:

help?> spacedirections
search: spacedirections                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
  spacedirections(img) -> (axis1, axis2, ...)                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                               
  Return a tuple-of-tuples, each axis[i] representing the displacement vector between adjacent pixels along spatial axis i of the image array, relative to some external coordinate system ("physical coordinates").                                                           
                                                                                                                                                                                                                                                                               
  By default this is computed from pixelspacing, but you can set this manually using ImagesMeta.                                                                                                                                                                               
                                                                                                                                                                                                                                                                               
  ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────  
                                                                                                                                                                                                                                                                               
  spacedirections(img)                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                               
  Using ImageMetadata, you can set this property manually. For example, you could indicate that a photograph was taken with the camera tilted 30-degree relative to vertical using

  img["spacedirections"] = ((0.866025,-0.5),(0.5,0.866025))

  If not specified, it will be computed from pixelspacing(img), placing the spacing along the "diagonal". If desired, you can set this property in terms of physical units, and each axis can have distinct units.

[Tokazama]

It looks like a lot of the structure for a standardized format is in place, but I'm not entirely sure of the right way to go about conforming to this. This has been my thought process:

1. I want to be able to read an image into a format that is compatible with the entire "Image" library. However, I don't want to require the entire library in order to read my image into a compatible format (e.g. I shouldn't need "ImageFiltering" to read in my image but I want it to be compatible with its functions).
2. So I want my FileIO integrated package to depend on some standard that will predictably work with the beautiful "Image" library. I have "ImageMetadata" to provide the format but need "ImageCore" to provide the standard API. I'm not really sure where "ImageAxis" fits into this yet because my understanding of the documentation makes it seem really similar to "AxisArrays".
3. I make ImageMeta the final import type and write from ImageMeta type with the appropriate "fileformat" property. I have some checks for other appropriate properties in the writing step.

I'm sure most of what's necessary to move forward is in place and additional clarity will arise as more unique imaging formats are implemented. However, it's not entirely clear what is necessary to be Image library "compliant" or if that's even supposed to be a thing.

[timholy]

ImageCore = conversions/reinterpretations of values + frequently-used traits ImageAxis = AxisArrays + trait specializations for AxisArrays ImageMetadata = ImageAxis + addition info, including additional trait specializations

You shouldn't need anything higher in the stack than this. Here are ImageMetadata's requirements. No ImageFiltering/Images/etc.

[tknopp]

@Tokazama: Tim has designed Images "version 2" two years ago exactly in such a way that the dependencies are minimalistic. Don't get blended from the number of packages involved, this is more a sign of good modularity. Actually this is why I want to push the discussion into making "ImageMetadata" standard format for storing complex files like nifti and dicom.

I don't have the bandwidth to contribute much code right now but it would be absolutely awesome if you could play with the idea for NiFTi.jl. I can give you rights such that you can work on a branch.

For the record, here (https://github.com/MagneticParticleImaging/MPIFiles.jl/blob/master/src/Image.jl) you can find some code where I have the correspondence (ImageMetadata <-> MDF File). MDFs is a domain specific image format that I have developed.

[Tokazama]

I'd be happy to put a bit of time into that.

[tknopp]

@Tokazama you should now have rights to make changes to this package. Best thing for new ideas is creating a branch and working there.

[Tokazama]

tldr: I'm working on this again. It's not ready yet but it should be ready for heavy use very soon.

Apologies for the long wait on this. Classes and projects put this on hold for a while. Here are some of the major changes in the most recent update and what the reasoning behind them were.

1. Generic backend for image reading (ImageFormats): Once all the header information is in place the ImageStream hands off the process to ImageFormats. I'm not sure whether this ends up being useful for other packages yet, but the implementation seems basic enough to use in other code bases without much effort.
2. Dropped theNiftiHeader types: The use of ImageFormats means that once all the info is available in an ImageStream there is no use for a NIftiHeader. It would only serve as an intermediate container. The one downside to this is that it may make readability of code a bit harder. I'm trying to make this as clear as possible in the code itself so others can tell what the basic outline of the header is still.
3. Writing files is not yet resolved so use at your own caution. Hopefully I'll figure this out today or tomorrow.
4. Transforms and sinks: I decided on going forward with ImageFormats because the NIfTI standard has a stupid amount of data types it tries to support. I've been doing some research on it and have found out that many of them are in fact used but sometimes by one or two programs out in the wild. This is unfortunately the nature of the beast. Something like load(file, sink::AbstractArray) becomes very attractive in this situation. Unfortunately I only have limited support for this now. I'm going to be looking at NamedDims.jl for simplifying movement between bits types and structs (like RGB, RGBA, Quaternion, Triangle, Point, tons of stats distributions,...). This will just take the dimension names (or axisnames if you prefer) and permutedims and convert to the proper element type (like channelview and colorview)

I still need to put together proper tests and fix a lot of the documentation, but I should actually be able to work on this again now that I have a bit more time.

[tknopp]

Great progress. Isn't NamesDims.jl just an alternative to AxisArrays.jl? The later also encodes the dim name with zero cost abstraction.

I am currently using NIfTI.jl here https://github.com/MagneticResonanceImaging/MRIReco.jl/blob/master/src/IO/Nifti.jl#L9 in my MRI reconstruction package. There I need to create a NIfTI file "from scratch" although it possible to retrieve some additional metadata from the underlying ISMRMRD raw data file. Once you have proper write support, I am looking forward learning the interface how to write the NIFTI file.

[Tokazama]

NamedDims.jl is supposed to be the future implementation of NamesDims.jl and have zero cost at compile time (see https://github.com/JuliaCollections/AxisArraysFuture/issues/1#issuecomment-484702271). I'm not sure if this will happen anytime soon, but the functionality I'd like to get out of it isn't necessary for most use cases, so that's a minor issue for now.

The goal is to have a generic interface for any AbstractArray so that niwrite(::String, ::AbstractArray) just works. The orientation and axes are acquired using the ImageMeta interface. Looking at your link you could probably just wrap niwrite in saveImage.

[tknopp]

Ok, I missed that new direction for AxisArrays. Whatever is used in the end. Most important is that we agree on a single standard, so that this remains compatible throughout different packages.

Of what type should the AbstractArray be if I want to provide some control over how the data is stored? should / can ist be an ImageMeta that wraps an AxisArray? Will this store the pixelspacing and image center correctly? Then I would be interested how to report the orientation in the ImageMeta.

[Tokazama]

Yeah, ImageMeta{T,N,<:AxisArray} is probably the way to go. I'm trying to use the same syntax that is used in Images. I figure that no matter how AxisArrays develops in the future that working around these standard functions will be easiest. The orientation is acquired by calling spacedirectionsfrom ImageCore.

[Tokazama]

I figured there needed to be an update on this as my progress has been embarrassingly slow. Fortunately, I mostly just have to write a couple papers and work on this over the next two weeks. I actually was using my branch pretty heavily at one point (feeding thousands of images into a Flux model), so I don't expect there to be any major issues for people trying to do their day to day stuff. I've also enlisted a physics guy to stress test it with some multi-gigabyte DTI data he's working on. Hopefully, this will cover any major blind spots I may have.

I'm going go over my plan moving forward on this as it appears some people may be waiting for me to actually get into gear and do what I actually said I'd do with this :P. I've ordered it according to priority so others may have an idea of when to expect things.

1. Basic I/O support using FilIO syntax
   • [X] NIfTI-1 support
   • [X] NIfTI-2 support
   • [ ] Prepare for multi-threading: there are some points in the code that need to be refactored so multi-threading is easier to implement for the approaching 1.3 release.
   • [ ] Implement Brain imaging data structure support (likely in a different package that this could depend on)
     • It turns out file extensions are very important for NIfTI. BIDS seems to be the most sensible approach to doing this because it relies heavily on file paths and extensions to determine organization. Therefore, it can assist in identifying what format a file should be loaded as before parsing the entire header.
     • This will also help standardize the metadata that shoud be kept around and that needs some sort of designated syntax to access.
   • [ ] Documentation
     • [ ] Method documentation: There's a lot of documentation I have floating around on this but it's a mess and needs to be supplemented.
     • [ ] Examples: It would be nice to have some examples where we demonstrate how to straightforward it is to load an image and do some traditional processing. I was working on deriving the various tensors from a DWI image at one point but I may just have to do a simulated mass univariate analysis.
2. Support extensions to the NIfTI specification
   • [ ] CIfTI: Used commonly in the Human Connectome Project which I'm currently working with and need a way to get my data into Julia.
   • [ ] GIfTI: Very similar implementation to CIfTI but with some slight differences, so work on these two should progress together closely.
3. Visualizations/plotting
   • [ ] There are currently methods that aren't too difficult for perusing images by slices. However, It would be very useful to have mesh plots for looking at regions of interest. Some of these CIFTI/GIFTI types also use point clouds and triangle surfaces.

I expect to finish the first one fairly soon (weeks). I have no time table for the second or third one but I need them for several other projects that are coming up this next month. So if they don't get done soon that there will be some alternative strategy that fulfills a similar role.

[tknopp]

Hey, @Tokazama. Maybe it makes sense to chunk this a bit. I think 1. + some minimal form of documentation would be most important since it's the interface that people will rely on. Multithreading, 2. and 3. could perfectly follow afterwards.

If you are fine with the interface, tests are passing just go ahead merge and release.

[Tokazama]

I agree. I want to get the first out the door ASAP. I'm putting together the BIDS stuff here for now. Right now I'm just outlining everything so that it doesn't have to be rewritten every time a new format is supported. I don't plan to actual finish any more than the MRI relevant specification before getting the NIfTI branch ready to go.

[timholy]

It turns out file extensions are very important for NIfTI

I am sure you have seen this. If it helps, you could add a detection function to FileIO and then use its query interface.

[Tokazama]

Yes I have. And that's useful for the vast majority of cases, but there's a whole slew of CIfTI formats that require "somename.unique_variant.nii" so it requires an additional parsing of the "unique_variant" portion in addition to detecting the "nii". This also tells us it's likely a NIfTI-2 format because CIfTI is exclusively NIfTI-2 format. There are also unique combinations where if you have a certain set of extension then you expect to have another extension type file in the same folder (kind of like how Analyze requires a ".hdr" and ".img").

It's not something that will hold me up on getting the first release out the door too much, but I need to at least have a logical way to change how header data is handled based on the file extension. Especially the "extension" of the NIfTI file between the raw data and header. That stuff get's a little crazy.

[Tokazama]

BTW. My latest stuff is nearly ready to be merged to my branch and once those tests are passing I'll merge. The only hold up at this point is getting NamedDims, AxisArrays, etc. compatibility finalized.

[timholy]

The only hold up at this point is getting NamedDims, AxisArrays, etc. compatibility finalized.

So we can look forward to merging in 2021? :smile: I should engage a bit more with that process, it's clearly important. So little time though...

[Tokazama]

I'm finishing up a pretty thorough write up on how to move that ahead that I'll post to relevant channels (hopefully today). I have to finish a fairly important analysis today, but once that's finished I'll push stuff to my branch. It probably won't work outside of my local packages without a the array stuff in place, but at least people could take a look at the method names and syntax so that we're sure they gel right with everyone.