jamesdolezal
multimodal inputs of clinical information in multiple instance learning
Feature
Multimodal inputs like clinical information in Multiple-instance learning
Pitch
When doing multiple-instance learning, as far as I read GitHub and the document, it seems we cannot use clinical information as input. I would appreciate it if you could let me know how I can conduct such kinds of calculations.
"Multimodal models In addition to training using image data, clinical data can also be provided as model input by passing annotation column headers to the variable input_header. This input is concatenated at the post-convolutional layer, prior to any configured hidden layers. If desired, models can also be trained with clinical input data alone, without images, by using the hyperparameter argument drop_images=True."
https://slideflow.dev/training/
Per the github, this is included - I have not used it yet. Hope this helps.
Thanks for suggesting this enhancement. At present, we have out-of-the-box support for multi-modal training (using clinical variables from the annotations file) for tile-based models only.
In version 2.3, we added a mm_attention_mil model that is designed to support multi-magnification input, by providing multiple bags (of arbitrary dimension) for each slide (https://slideflow.dev/mil/#multi-magnification-mil). In this design, each input modality has a separate attention head, and the attention-reduced feature vectors are reduced prior to being passed to the fully-connected layer at the end of the network.
Once could extend this existing multi-magnification functionality for arbitrary multi-modal support (e.g. clinical factors, gene expression, spatial transcriptomics, etc). To accomplish this, you'd need to generate bags for the new modality, then pass the folder of bags to the train_mil() function as described above.
With that said, it would be nice if Slideflow could perform this automatically to streamline easy multi-modal models trained on clinical data. We'd also like to be able to support muti-magnification & multimodal training for more architectures than just Attention_MIL. We are prioritizing development of more robust multimodal training tools for MIL and anticipate inclusion of a new toolset for this in version 2.5.
