Project-MONAI
Expanded Discussion of Tensor Type/Format/Modality in Bundle Definition
I have proposed some additions to how model input and output tensors are specified in PR https://github.com/Project-MONAI/MONAI/pull/4047. The idea is that the type of a tensor states if it's an image, a value sequence like a signal, sets of tuple values like points, etc. The format states the content of what is stored in one of these. Modality is a part of the format specification which states what subtype of a format is stored, so for MR this would the acquisition protocol like T1, for others it can whatever is appropriate.
The purpose is to mode accurately describe what the inputs and outputs mean so that they can be visualized and other handling operations automatically inferred. I had intended the definitions to cover a range of non-image data as well which we intend to support going forward.
The question is then what is missing from this proposed specification extension, what sort of other formats should there be, and how can this information be presented or modified to help in your particular use case.
We may need to add a generic "features" (in the ML sense of the word) category for vectors of non-image input data such as laboratory values, demographic information, etc, that a model may use in addition to an image when making a prediction?
